[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:20:10 dist-test-slave-4bk3 krb5kdc[3926](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:20:10 dist-test-slave-4bk3 krb5kdc[3926](info): set up 2 sockets
Sep 25 15:20:10 dist-test-slave-4bk3 krb5kdc[3926](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 15:20:11.664878 3919 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Sep 25 15:20:11 dist-test-slave-4bk3 krb5kdc[3926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:20:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:20:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.211.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
I20260925 15:20:11.795881 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:37159
--webserver_interface=127.3.211.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:39519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.211.254:37159
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:20:11.882228 3939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:11.882531 3939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:11.882573 3939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:11.883898 3939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:20:11.883982 3939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:11.884006 3939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:11.884027 3939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:20:11.884057 3939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:20:11.885910 3939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:39519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.211.254:37159
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:37159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.211.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:11.886345 3939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:11.886636 3939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:11.892576 3945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:11.892647 3939 server_base.cc:1034] running on GCE node
W20260925 15:20:11.893220 3944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:11.893477 3947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:11.893837 3939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:11.894255 3939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:11.895411 3939 hybrid_clock.cc:648] HybridClock initialized: now 1790349611895388 us; error 46 us; skew 500 ppm
Sep 25 15:20:11 dist-test-slave-4bk3 krb5kdc[3926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349611, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:11.902994 3939 init.cc:377] Logged in from keytab as kudu/127.3.211.254@KRBTEST.COM (short username kudu)
I20260925 15:20:11.903733 3939 webserver.cc:460] Webserver started at http://127.3.211.254:42143/ using document root <none> and password file <none>
I20260925 15:20:11.904052 3939 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:11.904104 3939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:11.904278 3939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:11.905645 3939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/master-0/data/instance:
uuid: "b4f57d5d7e4b490aaa9d079974082614"
format_stamp: "Formatted at 2026-09-25 15:20:11 on dist-test-slave-4bk3"
server_key: "4ae7ad347afa8b744b39439420816ddc"
server_key_iv: "dc5853f7fdd1069c09cd2852246f7aa0"
server_key_version: "encryptionkey@0"
I20260925 15:20:11.906248 3939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/master-0/wal/instance:
uuid: "b4f57d5d7e4b490aaa9d079974082614"
format_stamp: "Formatted at 2026-09-25 15:20:11 on dist-test-slave-4bk3"
server_key: "4ae7ad347afa8b744b39439420816ddc"
server_key_iv: "dc5853f7fdd1069c09cd2852246f7aa0"
server_key_version: "encryptionkey@0"
I20260925 15:20:11.907980 3939 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:20:11.909677 3954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:11.910252 3939 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:11.910422 3939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/master-0/wal
uuid: "b4f57d5d7e4b490aaa9d079974082614"
format_stamp: "Formatted at 2026-09-25 15:20:11 on dist-test-slave-4bk3"
server_key: "4ae7ad347afa8b744b39439420816ddc"
server_key_iv: "dc5853f7fdd1069c09cd2852246f7aa0"
server_key_version: "encryptionkey@0"
I20260925 15:20:11.910590 3939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:11.923086 3939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:11.925829 3939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:11.926018 3939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:11.931033 3939 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.254:37159
I20260925 15:20:11.931083 4006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.254:37159 every 8 connection(s)
I20260925 15:20:11.931586 3939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
I20260925 15:20:11.932127 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 3939
I20260925 15:20:11.932224 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/master-0/wal/instance
I20260925 15:20:11.932754 4007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:11.932927 3919 external_mini_cluster.cc:1428] Setting key 60cd871e50d0a15e611369be0aab47f6
I20260925 15:20:11.935711 4007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4f57d5d7e4b490aaa9d079974082614: Bootstrap starting.
I20260925 15:20:11.936314 4007 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4f57d5d7e4b490aaa9d079974082614: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:11.936645 4007 log.cc:824] T 00000000000000000000000000000000 P b4f57d5d7e4b490aaa9d079974082614: Log is configured to *not* fsync() on all Append() calls
I20260925 15:20:11.937352 4007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4f57d5d7e4b490aaa9d079974082614: No bootstrap required, opened a new log
I20260925 15:20:11.939484 4007 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4f57d5d7e4b490aaa9d079974082614 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4f57d5d7e4b490aaa9d079974082614" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 37159 } }
I20260925 15:20:11.939674 4007 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4f57d5d7e4b490aaa9d079974082614 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:11.939733 4007 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4f57d5d7e4b490aaa9d079974082614 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4f57d5d7e4b490aaa9d079974082614, State: Initialized, Role: FOLLOWER
I20260925 15:20:11.939936 4007 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4f57d5d7e4b490aaa9d079974082614 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4f57d5d7e4b490aaa9d079974082614" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 37159 } }
I20260925 15:20:11.940052 4007 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4f57d5d7e4b490aaa9d079974082614 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:20:11.940099 4007 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4f57d5d7e4b490aaa9d079974082614 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:20:11.940155 4007 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4f57d5d7e4b490aaa9d079974082614 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:11.940867 4007 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4f57d5d7e4b490aaa9d079974082614 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4f57d5d7e4b490aaa9d079974082614" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 37159 } }
I20260925 15:20:11.941012 4007 leader_election.cc:302] T 00000000000000000000000000000000 P b4f57d5d7e4b490aaa9d079974082614 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4f57d5d7e4b490aaa9d079974082614; no voters:
I20260925 15:20:11.941248 4007 leader_election.cc:288] T 00000000000000000000000000000000 P b4f57d5d7e4b490aaa9d079974082614 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:20:11.941269 4012 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4f57d5d7e4b490aaa9d079974082614 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:20:11.941386 4012 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4f57d5d7e4b490aaa9d079974082614 [term 1 LEADER]: Becoming Leader. State: Replica: b4f57d5d7e4b490aaa9d079974082614, State: Running, Role: LEADER
I20260925 15:20:11.941463 4012 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4f57d5d7e4b490aaa9d079974082614 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4f57d5d7e4b490aaa9d079974082614" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 37159 } }
I20260925 15:20:11.941553 4007 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4f57d5d7e4b490aaa9d079974082614 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:20:11.941826 4013 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4f57d5d7e4b490aaa9d079974082614 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4f57d5d7e4b490aaa9d079974082614" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4f57d5d7e4b490aaa9d079974082614" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 37159 } } }
I20260925 15:20:11.941946 4013 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4f57d5d7e4b490aaa9d079974082614 [sys.catalog]: This master's current role is: LEADER
I20260925 15:20:11.942133 4014 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4f57d5d7e4b490aaa9d079974082614 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4f57d5d7e4b490aaa9d079974082614. Latest consensus state: current_term: 1 leader_uuid: "b4f57d5d7e4b490aaa9d079974082614" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4f57d5d7e4b490aaa9d079974082614" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 37159 } } }
I20260925 15:20:11.942227 4014 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4f57d5d7e4b490aaa9d079974082614 [sys.catalog]: This master's current role is: LEADER
I20260925 15:20:11.942359 4017 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:20:11.943163 4017 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:20:11.945379 4017 catalog_manager.cc:1347] Generated new cluster ID: f8cccfbd57da4a44b7bd41c24840ee12
I20260925 15:20:11.945447 4017 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 15:20:11 dist-test-slave-4bk3 krb5kdc[3926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349611, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:20:11.952865 4009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:11.933897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48748 (local address 127.3.211.254:37159)
0925 15:20:11.934112 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:11.934120 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:11.934422 (+ 302us) server_negotiation.cc:407] Connection header received
0925 15:20:11.936706 (+ 2284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:11.936713 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:11.936878 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:11.937033 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:11.938928 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:11.940442 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:11.942331 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:11.943055 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:11.943157 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:11.948188 (+ 5031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:11.948194 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:11.948195 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:11.948210 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:11.950396 (+ 2186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:11.951098 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:11.951102 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:11.951102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:11.951234 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:11.951590 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:11.951591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:11.951591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:11.951913 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:11.952087 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:11.952477 (+ 390us) server_negotiation.cc:299] Negotiation successful
0925 15:20:11.952636 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":86,"threads_started":1}
I20260925 15:20:11.958135 4017 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:20:11.958815 4017 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:20:11.964681 4017 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4f57d5d7e4b490aaa9d079974082614: Generated new TSK 0
I20260925 15:20:11.964916 4017 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.211.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab.
Entry for principal kudu/127.3.211.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab.
I20260925 15:20:12.001639 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.193:0
--local_ip_for_outbound_sockets=127.3.211.193
--webserver_interface=127.3.211.193
--webserver_port=0
--tserver_master_addrs=127.3.211.254:37159
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab
--principal=kudu/127.3.211.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:39519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:20:12.091527 4033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:12.091768 4033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:12.091795 4033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:12.093173 4033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:12.093236 4033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:12.093268 4033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.193
I20260925 15:20:12.094885 4033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:39519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab
--principal=kudu/127.3.211.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.211.193
--webserver_port=0
--tserver_master_addrs=127.3.211.254:37159
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.193
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:12.095166 4033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:12.095444 4033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:12.100946 4039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:12.101022 4033 server_base.cc:1034] running on GCE node
W20260925 15:20:12.101091 4038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:12.101032 4041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:12.101629 4033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:12.101977 4033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:12.103186 4033 hybrid_clock.cc:648] HybridClock initialized: now 1790349612103160 us; error 49 us; skew 500 ppm
Sep 25 15:20:12 dist-test-slave-4bk3 krb5kdc[3926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349612, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:12.106272 4033 init.cc:377] Logged in from keytab as kudu/127.3.211.193@KRBTEST.COM (short username kudu)
I20260925 15:20:12.106813 4033 webserver.cc:460] Webserver started at http://127.3.211.193:35501/ using document root <none> and password file <none>
I20260925 15:20:12.107061 4033 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:12.107133 4033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:12.107287 4033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:12.108469 4033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/ts-0/data/instance:
uuid: "0a8108fee1ed46ac9812cc7adaa6ed29"
format_stamp: "Formatted at 2026-09-25 15:20:12 on dist-test-slave-4bk3"
server_key: "26c3fa50e2dafafb6f846c3b5f46489b"
server_key_iv: "9d4e76182ac54819e3ffd2c7b1b8e693"
server_key_version: "encryptionkey@0"
I20260925 15:20:12.108848 4033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/ts-0/wal/instance:
uuid: "0a8108fee1ed46ac9812cc7adaa6ed29"
format_stamp: "Formatted at 2026-09-25 15:20:12 on dist-test-slave-4bk3"
server_key: "26c3fa50e2dafafb6f846c3b5f46489b"
server_key_iv: "9d4e76182ac54819e3ffd2c7b1b8e693"
server_key_version: "encryptionkey@0"
I20260925 15:20:12.110328 4033 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:20:12.111176 4048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:12.111407 4033 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:12.111521 4033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/ts-0/wal
uuid: "0a8108fee1ed46ac9812cc7adaa6ed29"
format_stamp: "Formatted at 2026-09-25 15:20:12 on dist-test-slave-4bk3"
server_key: "26c3fa50e2dafafb6f846c3b5f46489b"
server_key_iv: "9d4e76182ac54819e3ffd2c7b1b8e693"
server_key_version: "encryptionkey@0"
I20260925 15:20:12.111613 4033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:12.130599 4033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:12.132978 4033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:12.133158 4033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:12.133428 4033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:20:12.133765 4033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:20:12.133823 4033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:12.133867 4033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:20:12.133896 4033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:12.140926 4033 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.193:40441
I20260925 15:20:12.141006 4161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.193:40441 every 8 connection(s)
I20260925 15:20:12.141386 4033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:20:12.147536 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 4033
I20260925 15:20:12.147660 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/ts-0/wal/instance
I20260925 15:20:12.147850 3919 external_mini_cluster.cc:1428] Setting key 0ce9d07ac8f0d0d145ae4611756c62b1
Sep 25 15:20:12 dist-test-slave-4bk3 krb5kdc[3926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349612, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.193@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:20:12.153481 4009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:12.142404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:55875 (local address 127.3.211.254:37159)
0925 15:20:12.142477 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:12.142479 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:12.142705 (+ 226us) server_negotiation.cc:407] Connection header received
0925 15:20:12.143933 (+ 1228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:12.143934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:12.143948 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:12.144027 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:12.144804 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:12.145331 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:12.146437 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:12.146676 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:12.146749 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:12.149976 (+ 3227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:12.149978 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:12.149978 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:12.149988 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:12.151703 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:12.152257 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:12.152259 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:12.152259 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:12.152384 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:12.152691 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:12.152692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:12.152692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:12.152835 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:12.152922 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:12.153251 (+ 329us) server_negotiation.cc:299] Negotiation successful
0925 15:20:12.153369 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 15:20:12.153702 4164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:12.142385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:37159 (local address 127.3.211.193:55875)
0925 15:20:12.142546 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:12.142609 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:12.143790 (+ 1181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:12.144089 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:12.144092 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:12.144460 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:12.144679 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:12.144683 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:12.145466 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:12.145468 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:12.146325 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:12.146327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:12.146872 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:12.146874 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:12.147083 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:12.148051 (+ 968us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:12.148065 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:12.149809 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:12.151840 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:12.151853 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:12.151856 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:12.152145 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:12.152471 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:12.152472 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:12.152472 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:12.152612 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:12.152910 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:12.152918 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:12.153088 (+ 170us) client_negotiation.cc:769] Sending connection context
0925 15:20:12.153227 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 15:20:12.153416 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":87,"threads_started":1}
I20260925 15:20:12.154289 4162 heartbeater.cc:347] Connected to a master server at 127.3.211.254:37159
I20260925 15:20:12.154422 4162 heartbeater.cc:464] Registering TS with master...
I20260925 15:20:12.154680 4162 heartbeater.cc:511] Master 127.3.211.254:37159 requested a full tablet report, sending...
I20260925 15:20:12.155292 3971 ts_manager.cc:194] Registered new tserver with Master: 0a8108fee1ed46ac9812cc7adaa6ed29 (127.3.211.193:40441)
I20260925 15:20:12.156051 3971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.193@KRBTEST.COM'} at 127.3.211.193:55875
WARNING: no policy specified for kudu/127.3.211.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab.
Entry for principal kudu/127.3.211.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab.
I20260925 15:20:12.177686 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.194:0
--local_ip_for_outbound_sockets=127.3.211.194
--webserver_interface=127.3.211.194
--webserver_port=0
--tserver_master_addrs=127.3.211.254:37159
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab
--principal=kudu/127.3.211.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:39519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:20:12.263617 4167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:12.263849 4167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:12.263876 4167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:12.265093 4167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:12.265136 4167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:12.265167 4167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.194
I20260925 15:20:12.266921 4167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:39519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab
--principal=kudu/127.3.211.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.211.194
--webserver_port=0
--tserver_master_addrs=127.3.211.254:37159
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.194
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:12.267164 4167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:12.267359 4167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:12.272987 4173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:12.272992 4172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:12.272992 4175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:12.273216 4167 server_base.cc:1034] running on GCE node
I20260925 15:20:12.273412 4167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:12.273636 4167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:12.274804 4167 hybrid_clock.cc:648] HybridClock initialized: now 1790349612274789 us; error 46 us; skew 500 ppm
Sep 25 15:20:12 dist-test-slave-4bk3 krb5kdc[3926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349612, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:12.277544 4167 init.cc:377] Logged in from keytab as kudu/127.3.211.194@KRBTEST.COM (short username kudu)
I20260925 15:20:12.278164 4167 webserver.cc:460] Webserver started at http://127.3.211.194:46115/ using document root <none> and password file <none>
I20260925 15:20:12.278450 4167 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:12.278533 4167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:12.278705 4167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:12.279870 4167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/ts-1/data/instance:
uuid: "6e2d44565ba744c89338dd6d922a955b"
format_stamp: "Formatted at 2026-09-25 15:20:12 on dist-test-slave-4bk3"
server_key: "39d5384a2ae94e75036121b70e1977b5"
server_key_iv: "79e463c4212ae00f4f695d0aab31c635"
server_key_version: "encryptionkey@0"
I20260925 15:20:12.280272 4167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/ts-1/wal/instance:
uuid: "6e2d44565ba744c89338dd6d922a955b"
format_stamp: "Formatted at 2026-09-25 15:20:12 on dist-test-slave-4bk3"
server_key: "39d5384a2ae94e75036121b70e1977b5"
server_key_iv: "79e463c4212ae00f4f695d0aab31c635"
server_key_version: "encryptionkey@0"
I20260925 15:20:12.282052 4167 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260925 15:20:12.283071 4182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:12.283452 4167 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:12.283603 4167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/ts-1/wal
uuid: "6e2d44565ba744c89338dd6d922a955b"
format_stamp: "Formatted at 2026-09-25 15:20:12 on dist-test-slave-4bk3"
server_key: "39d5384a2ae94e75036121b70e1977b5"
server_key_iv: "79e463c4212ae00f4f695d0aab31c635"
server_key_version: "encryptionkey@0"
I20260925 15:20:12.283710 4167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:12.300971 4167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:12.303596 4167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:12.303795 4167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:12.304059 4167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:20:12.304473 4167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:20:12.304529 4167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:12.304574 4167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:20:12.304600 4167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:12.310871 4167 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.194:36781
I20260925 15:20:12.310956 4295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.194:36781 every 8 connection(s)
I20260925 15:20:12.311302 4167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
I20260925 15:20:12.313601 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 4167
I20260925 15:20:12.313720 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/ts-1/wal/instance
I20260925 15:20:12.313905 3919 external_mini_cluster.cc:1428] Setting key 13ff126000c3645f294b0b9d24335d9f
Sep 25 15:20:12 dist-test-slave-4bk3 krb5kdc[3926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349612, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.194@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.211.195@KRBTEST.COM; defaulting to no policy
I20260925 15:20:12.323431 4009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:12.312191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:46015 (local address 127.3.211.254:37159)
0925 15:20:12.312346 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:12.312349 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:12.312571 (+ 222us) server_negotiation.cc:407] Connection header received
0925 15:20:12.313147 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:12.313149 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:12.313161 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:12.313215 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:12.314683 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:12.315297 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:12.316740 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:12.316948 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:12.317055 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:12.319848 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:12.319850 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:12.319850 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:12.319859 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:12.321467 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:12.322119 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:12.322122 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:12.322122 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:12.322253 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:12.322678 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:12.322680 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:12.322680 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:12.322893 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:12.322973 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:12.323279 (+ 306us) server_negotiation.cc:299] Negotiation successful
0925 15:20:12.323359 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 15:20:12.323617 4298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:12.312154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:37159 (local address 127.3.211.194:46015)
0925 15:20:12.312394 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:12.312441 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:12.313032 (+ 591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:12.313290 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:12.313295 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:12.314296 (+ 1001us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:12.314533 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:12.314538 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:12.315401 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:12.315402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:12.316639 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:12.316642 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:12.317082 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:12.317083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:12.317213 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:12.317937 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:12.317950 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:12.319705 (+ 1755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:12.321584 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:12.321595 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:12.321598 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:12.321954 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:12.322380 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:12.322381 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:12.322382 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:12.322549 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:12.322983 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:12.322990 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:12.323147 (+ 157us) client_negotiation.cc:769] Sending connection context
0925 15:20:12.323248 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 15:20:12.323387 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":100,"threads_started":1}
I20260925 15:20:12.324175 4296 heartbeater.cc:347] Connected to a master server at 127.3.211.254:37159
I20260925 15:20:12.324268 4296 heartbeater.cc:464] Registering TS with master...
I20260925 15:20:12.324479 4296 heartbeater.cc:511] Master 127.3.211.254:37159 requested a full tablet report, sending...
I20260925 15:20:12.324877 3971 ts_manager.cc:194] Registered new tserver with Master: 6e2d44565ba744c89338dd6d922a955b (127.3.211.194:36781)
I20260925 15:20:12.325224 3971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.194@KRBTEST.COM'} at 127.3.211.194:46015
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab.
Entry for principal kudu/127.3.211.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab.
I20260925 15:20:12.343748 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.195:0
--local_ip_for_outbound_sockets=127.3.211.195
--webserver_interface=127.3.211.195
--webserver_port=0
--tserver_master_addrs=127.3.211.254:37159
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab
--principal=kudu/127.3.211.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:39519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:20:12.428781 4301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:12.429035 4301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:12.429080 4301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:12.430370 4301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:12.430451 4301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:12.430502 4301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.195
I20260925 15:20:12.432175 4301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:39519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab
--principal=kudu/127.3.211.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.211.195
--webserver_port=0
--tserver_master_addrs=127.3.211.254:37159
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.195
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:12.432515 4301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:12.432786 4301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:12.438248 4306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:12.438248 4309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:12.438338 4307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:12.438839 4301 server_base.cc:1034] running on GCE node
I20260925 15:20:12.439074 4301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:12.439347 4301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:12.440526 4301 hybrid_clock.cc:648] HybridClock initialized: now 1790349612440509 us; error 47 us; skew 500 ppm
Sep 25 15:20:12 dist-test-slave-4bk3 krb5kdc[3926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349612, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:12.443745 4301 init.cc:377] Logged in from keytab as kudu/127.3.211.195@KRBTEST.COM (short username kudu)
I20260925 15:20:12.444258 4301 webserver.cc:460] Webserver started at http://127.3.211.195:46361/ using document root <none> and password file <none>
I20260925 15:20:12.444553 4301 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:12.444613 4301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:12.444715 4301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:12.445920 4301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/ts-2/data/instance:
uuid: "bcdc43656a0d4e84aa9cb79e7c8b8266"
format_stamp: "Formatted at 2026-09-25 15:20:12 on dist-test-slave-4bk3"
server_key: "d15da03c81b2dc73e3e18791b26f1300"
server_key_iv: "87e013d85aba5e7fe191cb0a9deec2cc"
server_key_version: "encryptionkey@0"
I20260925 15:20:12.446244 4301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/ts-2/wal/instance:
uuid: "bcdc43656a0d4e84aa9cb79e7c8b8266"
format_stamp: "Formatted at 2026-09-25 15:20:12 on dist-test-slave-4bk3"
server_key: "d15da03c81b2dc73e3e18791b26f1300"
server_key_iv: "87e013d85aba5e7fe191cb0a9deec2cc"
server_key_version: "encryptionkey@0"
I20260925 15:20:12.447588 4301 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:20:12.448410 4316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:12.448587 4301 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:12.448683 4301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/ts-2/wal
uuid: "bcdc43656a0d4e84aa9cb79e7c8b8266"
format_stamp: "Formatted at 2026-09-25 15:20:12 on dist-test-slave-4bk3"
server_key: "d15da03c81b2dc73e3e18791b26f1300"
server_key_iv: "87e013d85aba5e7fe191cb0a9deec2cc"
server_key_version: "encryptionkey@0"
I20260925 15:20:12.448755 4301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:12.456928 4301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:12.460060 4301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:12.460191 4301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:12.460472 4301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:20:12.460783 4301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:20:12.460813 4301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:12.460839 4301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:20:12.460851 4301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:12.466830 4301 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.195:37427
I20260925 15:20:12.466912 4429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.195:37427 every 8 connection(s)
I20260925 15:20:12.467242 4301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
I20260925 15:20:12.469038 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 4301
I20260925 15:20:12.469179 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790349610595785-3919-0/minicluster-data/ts-2/wal/instance
I20260925 15:20:12.469352 3919 external_mini_cluster.cc:1428] Setting key fb778a16ab98f659c9cbadbb9845392a
Sep 25 15:20:12 dist-test-slave-4bk3 krb5kdc[3926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349612, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.195@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:20:12.478909 4009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:12.468158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:45737 (local address 127.3.211.254:37159)
0925 15:20:12.468246 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:12.468248 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:12.468463 (+ 215us) server_negotiation.cc:407] Connection header received
0925 15:20:12.469027 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:12.469028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:12.469040 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:12.469149 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:12.470621 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:12.471203 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:12.472886 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:12.473120 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:12.473202 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:12.475604 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:12.475606 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:12.475607 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:12.475614 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:12.477207 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:12.477838 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:12.477839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:12.477839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:12.477892 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:12.478132 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:12.478132 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:12.478133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:12.478241 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:12.478295 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:12.478748 (+ 453us) server_negotiation.cc:299] Negotiation successful
0925 15:20:12.478835 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 15:20:12.479082 4432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:12.468104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:37159 (local address 127.3.211.195:45737)
0925 15:20:12.468268 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:12.468313 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:12.468901 (+ 588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:12.469339 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:12.469343 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:12.470207 (+ 864us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:12.470453 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:12.470480 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:12.471399 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:12.471401 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:12.472762 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:12.472767 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:12.473251 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:12.473252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:12.473390 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:12.473964 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:12.473975 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:12.475463 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:12.477382 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:12.477397 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:12.477400 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:12.477754 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:12.477959 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:12.477960 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:12.477960 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:12.478076 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:12.478336 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:12.478342 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:12.478490 (+ 148us) client_negotiation.cc:769] Sending connection context
0925 15:20:12.478693 (+ 203us) client_negotiation.cc:240] Negotiation successful
0925 15:20:12.478842 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":88,"threads_started":1}
I20260925 15:20:12.479671 4430 heartbeater.cc:347] Connected to a master server at 127.3.211.254:37159
I20260925 15:20:12.479768 4430 heartbeater.cc:464] Registering TS with master...
I20260925 15:20:12.479918 4430 heartbeater.cc:511] Master 127.3.211.254:37159 requested a full tablet report, sending...
I20260925 15:20:12.480533 3971 ts_manager.cc:194] Registered new tserver with Master: bcdc43656a0d4e84aa9cb79e7c8b8266 (127.3.211.195:37427)
I20260925 15:20:12.480901 3971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.195@KRBTEST.COM'} at 127.3.211.195:45737
I20260925 15:20:12.481930 3919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 15:20:12 dist-test-slave-4bk3 krb5kdc[3926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349612, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 15:20:12 dist-test-slave-4bk3 krb5kdc[3926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349612, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.211.193@KRBTEST.COM
I20260925 15:20:12.507978 4437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:12.498994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55468 (local address 127.3.211.193:40441)
0925 15:20:12.499160 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:12.499165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:12.499176 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:20:12.499203 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:12.499205 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:12.499218 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:12.499305 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:12.500138 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:12.500990 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:12.501778 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:12.502128 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:12.502201 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:12.504917 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:12.504929 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:12.504930 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:12.504940 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:12.506650 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:12.507059 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:12.507062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:12.507063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:12.507127 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:12.507394 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:12.507396 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:12.507396 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:12.507557 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:12.507715 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:12.507819 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 15:20:12.507892 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":93,"threads_started":1}
W20260925 15:20:12.508466 4075 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55468
Sep 25 15:20:12 dist-test-slave-4bk3 krb5kdc[3926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 15:20:12 dist-test-slave-4bk3 krb5kdc[3926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.211.193@KRBTEST.COM
I20260925 15:20:12.533975 4437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:12.525630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55484 (local address 127.3.211.193:40441)
0925 15:20:12.525698 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:12.525700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:12.525799 (+ 99us) server_negotiation.cc:407] Connection header received
0925 15:20:12.525832 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:12.525833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:12.525845 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:12.525923 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:12.526508 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:12.527013 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:12.527820 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:12.528030 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:12.528099 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:12.531146 (+ 3047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:12.531148 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:12.531148 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:12.531158 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:12.532810 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:12.533220 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:12.533221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:12.533221 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:12.533277 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:12.533487 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:12.533488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:12.533488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:12.533610 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:12.533712 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:12.533805 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 15:20:12.533873 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 15:20:12.534772 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 4033
I20260925 15:20:12.540390 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 4167
I20260925 15:20:12.545603 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 4301
I20260925 15:20:12.550371 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 3939
2026-09-25T15:20:12Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1960 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:20:12 dist-test-slave-4bk3 krb5kdc[4444](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:20:12 dist-test-slave-4bk3 krb5kdc[4444](info): set up 2 sockets
Sep 25 15:20:12 dist-test-slave-4bk3 krb5kdc[4444](info): commencing operation
krb5kdc: starting...
W20260925 15:20:13.607904 3919 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Sep 25 15:20:13 dist-test-slave-4bk3 krb5kdc[4444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:20:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:20:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.211.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
I20260925 15:20:13.731986 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:42125
--webserver_interface=127.3.211.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:46835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.211.254:42125
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:20:13.819018 4457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:13.819253 4457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:13.819278 4457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:13.820547 4457 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:20:13.820598 4457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:13.820613 4457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:13.820626 4457 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:20:13.820638 4457 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:20:13.822541 4457 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:46835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.211.254:42125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:42125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.211.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:13.823089 4457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:13.823372 4457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:13.828722 4463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:13.828837 4457 server_base.cc:1034] running on GCE node
W20260925 15:20:13.828733 4465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:13.828722 4462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:13.829186 4457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:13.829550 4457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:13.830720 4457 hybrid_clock.cc:648] HybridClock initialized: now 1790349613830765 us; error 88 us; skew 500 ppm
Sep 25 15:20:13 dist-test-slave-4bk3 krb5kdc[4444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349613, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:13.833384 4457 init.cc:377] Logged in from keytab as kudu/127.3.211.254@KRBTEST.COM (short username kudu)
I20260925 15:20:13.833794 4457 webserver.cc:460] Webserver started at http://127.3.211.254:46757/ using document root <none> and password file <none>
I20260925 15:20:13.833986 4457 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:13.834028 4457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:13.834131 4457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:13.835173 4457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/master-0/data/instance:
uuid: "4736e80311524d2f8169ae7a969d4f82"
format_stamp: "Formatted at 2026-09-25 15:20:13 on dist-test-slave-4bk3"
server_key: "72074e8801c90151c42d99ed368b9db4"
server_key_iv: "b7fa9a4fdb089cddb80d1843498a294a"
server_key_version: "encryptionkey@0"
I20260925 15:20:13.835484 4457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/master-0/wal/instance:
uuid: "4736e80311524d2f8169ae7a969d4f82"
format_stamp: "Formatted at 2026-09-25 15:20:13 on dist-test-slave-4bk3"
server_key: "72074e8801c90151c42d99ed368b9db4"
server_key_iv: "b7fa9a4fdb089cddb80d1843498a294a"
server_key_version: "encryptionkey@0"
I20260925 15:20:13.836993 4457 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:20:13.837842 4472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:13.838086 4457 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:13.838205 4457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/master-0/wal
uuid: "4736e80311524d2f8169ae7a969d4f82"
format_stamp: "Formatted at 2026-09-25 15:20:13 on dist-test-slave-4bk3"
server_key: "72074e8801c90151c42d99ed368b9db4"
server_key_iv: "b7fa9a4fdb089cddb80d1843498a294a"
server_key_version: "encryptionkey@0"
I20260925 15:20:13.838294 4457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:13.858218 4457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:13.860553 4457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:13.860729 4457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:13.864908 4457 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.254:42125
I20260925 15:20:13.864943 4524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.254:42125 every 8 connection(s)
I20260925 15:20:13.865325 4457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
I20260925 15:20:13.866353 4525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:13.868194 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 4457
I20260925 15:20:13.868323 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/master-0/wal/instance
I20260925 15:20:13.868507 3919 external_mini_cluster.cc:1428] Setting key 582d64a22be32b7bee07b3c71ca1b79e
I20260925 15:20:13.869222 4525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4736e80311524d2f8169ae7a969d4f82: Bootstrap starting.
I20260925 15:20:13.870023 4525 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4736e80311524d2f8169ae7a969d4f82: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:13.870493 4525 log.cc:824] T 00000000000000000000000000000000 P 4736e80311524d2f8169ae7a969d4f82: Log is configured to *not* fsync() on all Append() calls
I20260925 15:20:13.871443 4525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4736e80311524d2f8169ae7a969d4f82: No bootstrap required, opened a new log
I20260925 15:20:13.873890 4525 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4736e80311524d2f8169ae7a969d4f82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4736e80311524d2f8169ae7a969d4f82" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 42125 } }
I20260925 15:20:13.874087 4525 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4736e80311524d2f8169ae7a969d4f82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:13.874122 4525 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4736e80311524d2f8169ae7a969d4f82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4736e80311524d2f8169ae7a969d4f82, State: Initialized, Role: FOLLOWER
I20260925 15:20:13.874251 4525 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4736e80311524d2f8169ae7a969d4f82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4736e80311524d2f8169ae7a969d4f82" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 42125 } }
I20260925 15:20:13.874325 4525 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4736e80311524d2f8169ae7a969d4f82 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:20:13.874358 4525 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4736e80311524d2f8169ae7a969d4f82 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:20:13.874409 4525 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4736e80311524d2f8169ae7a969d4f82 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:13.875274 4525 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4736e80311524d2f8169ae7a969d4f82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4736e80311524d2f8169ae7a969d4f82" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 42125 } }
I20260925 15:20:13.875375 4525 leader_election.cc:302] T 00000000000000000000000000000000 P 4736e80311524d2f8169ae7a969d4f82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4736e80311524d2f8169ae7a969d4f82; no voters:
I20260925 15:20:13.875713 4525 leader_election.cc:288] T 00000000000000000000000000000000 P 4736e80311524d2f8169ae7a969d4f82 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:20:13.875736 4530 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4736e80311524d2f8169ae7a969d4f82 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:20:13.875916 4530 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4736e80311524d2f8169ae7a969d4f82 [term 1 LEADER]: Becoming Leader. State: Replica: 4736e80311524d2f8169ae7a969d4f82, State: Running, Role: LEADER
I20260925 15:20:13.876006 4530 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4736e80311524d2f8169ae7a969d4f82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4736e80311524d2f8169ae7a969d4f82" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 42125 } }
I20260925 15:20:13.876075 4525 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4736e80311524d2f8169ae7a969d4f82 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:20:13.876406 4530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4736e80311524d2f8169ae7a969d4f82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4736e80311524d2f8169ae7a969d4f82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4736e80311524d2f8169ae7a969d4f82" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 42125 } } }
I20260925 15:20:13.876488 4530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4736e80311524d2f8169ae7a969d4f82 [sys.catalog]: This master's current role is: LEADER
I20260925 15:20:13.876646 4530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4736e80311524d2f8169ae7a969d4f82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4736e80311524d2f8169ae7a969d4f82. Latest consensus state: current_term: 1 leader_uuid: "4736e80311524d2f8169ae7a969d4f82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4736e80311524d2f8169ae7a969d4f82" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 42125 } } }
I20260925 15:20:13.876722 4530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4736e80311524d2f8169ae7a969d4f82 [sys.catalog]: This master's current role is: LEADER
I20260925 15:20:13.876955 4536 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 25 15:20:13 dist-test-slave-4bk3 krb5kdc[4444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:20:13.877489 4536 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:20:13.879302 4536 catalog_manager.cc:1347] Generated new cluster ID: 0e839a3d75b64c1594ff6097cb13920f
I20260925 15:20:13.879362 4536 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:20:13.882057 4528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:13.869230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52522 (local address 127.3.211.254:42125)
0925 15:20:13.869447 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:13.869456 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:13.869486 (+ 30us) server_negotiation.cc:407] Connection header received
0925 15:20:13.870281 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:13.870298 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:13.870382 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:13.870562 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:13.871484 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:13.872917 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:13.873759 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:13.874390 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:13.874505 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:13.877677 (+ 3172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:13.877687 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:13.877688 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:13.877701 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:13.880182 (+ 2481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:13.880715 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:13.880718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:13.880719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:13.880827 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:13.881081 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:13.881082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:13.881083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:13.881360 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:13.881545 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:13.881693 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 15:20:13.881854 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":97,"threads_started":1}
I20260925 15:20:13.907034 4536 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:20:13.907635 4536 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:20:13.911890 4536 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4736e80311524d2f8169ae7a969d4f82: Generated new TSK 0
I20260925 15:20:13.912114 4536 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.211.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab.
Entry for principal kudu/127.3.211.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab.
I20260925 15:20:13.948045 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.193:0
--local_ip_for_outbound_sockets=127.3.211.193
--webserver_interface=127.3.211.193
--webserver_port=0
--tserver_master_addrs=127.3.211.254:42125
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab
--principal=kudu/127.3.211.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:46835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:20:14.039425 4551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:14.039670 4551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:14.039700 4551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:14.041618 4551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:14.041685 4551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:14.041719 4551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.193
I20260925 15:20:14.043725 4551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:46835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab
--principal=kudu/127.3.211.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.211.193
--webserver_port=0
--tserver_master_addrs=127.3.211.254:42125
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.193
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:14.043967 4551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:14.044173 4551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:14.049700 4557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:14.049677 4556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:14.049830 4551 server_base.cc:1034] running on GCE node
W20260925 15:20:14.049676 4559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:14.050155 4551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:14.050467 4551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:14.051695 4551 hybrid_clock.cc:648] HybridClock initialized: now 1790349614051669 us; error 48 us; skew 500 ppm
Sep 25 15:20:14 dist-test-slave-4bk3 krb5kdc[4444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349614, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:14.054991 4551 init.cc:377] Logged in from keytab as kudu/127.3.211.193@KRBTEST.COM (short username kudu)
I20260925 15:20:14.055516 4551 webserver.cc:460] Webserver started at http://127.3.211.193:33239/ using document root <none> and password file <none>
I20260925 15:20:14.055781 4551 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:14.055858 4551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:14.056027 4551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:14.057330 4551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/ts-0/data/instance:
uuid: "6067fe3a9c574c6dab76783fd1e2b56b"
format_stamp: "Formatted at 2026-09-25 15:20:14 on dist-test-slave-4bk3"
server_key: "cbdff58b5621d6376af2cb09e08b9533"
server_key_iv: "6e622132d211f0e9b60230aabba929ae"
server_key_version: "encryptionkey@0"
I20260925 15:20:14.058108 4551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/ts-0/wal/instance:
uuid: "6067fe3a9c574c6dab76783fd1e2b56b"
format_stamp: "Formatted at 2026-09-25 15:20:14 on dist-test-slave-4bk3"
server_key: "cbdff58b5621d6376af2cb09e08b9533"
server_key_iv: "6e622132d211f0e9b60230aabba929ae"
server_key_version: "encryptionkey@0"
I20260925 15:20:14.059731 4551 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:20:14.060725 4566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:14.061121 4551 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:20:14.061517 4551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/ts-0/wal
uuid: "6067fe3a9c574c6dab76783fd1e2b56b"
format_stamp: "Formatted at 2026-09-25 15:20:14 on dist-test-slave-4bk3"
server_key: "cbdff58b5621d6376af2cb09e08b9533"
server_key_iv: "6e622132d211f0e9b60230aabba929ae"
server_key_version: "encryptionkey@0"
I20260925 15:20:14.061673 4551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:14.099305 4551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:14.101826 4551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:14.102041 4551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:14.102308 4551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:20:14.102659 4551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:20:14.102716 4551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:14.102759 4551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:20:14.102787 4551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:14.108745 4551 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.193:42085
I20260925 15:20:14.108839 4679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.193:42085 every 8 connection(s)
I20260925 15:20:14.109174 4551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:20:14.114472 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 4551
I20260925 15:20:14.114769 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/ts-0/wal/instance
I20260925 15:20:14.115029 3919 external_mini_cluster.cc:1428] Setting key e1f5dfa17c0bfc1d40d8e123caa1bf19
Sep 25 15:20:14 dist-test-slave-4bk3 krb5kdc[4444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349614, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.193@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:20:14.121227 4528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:14.110018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:35257 (local address 127.3.211.254:42125)
0925 15:20:14.110100 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:14.110102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:14.110242 (+ 140us) server_negotiation.cc:407] Connection header received
0925 15:20:14.111218 (+ 976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:14.111219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:14.111230 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:14.111288 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:14.111944 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:14.112536 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:14.113593 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:14.113809 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:14.113870 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:14.116705 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:14.116708 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:14.116708 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:14.116718 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:14.119084 (+ 2366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:14.119685 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:14.119686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:14.119687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:14.119753 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:14.120109 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:14.120110 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:14.120110 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:14.120260 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:14.120392 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:14.120971 (+ 579us) server_negotiation.cc:299] Negotiation successful
0925 15:20:14.121127 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 15:20:14.121243 4682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:14.109925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:42125 (local address 127.3.211.193:35257)
0925 15:20:14.110085 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:14.110133 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:14.111095 (+ 962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:14.111359 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:14.111362 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:14.111646 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:14.111844 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:14.111848 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:14.112668 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:14.112669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:14.113503 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:14.113505 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:14.113913 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:14.113914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:14.114043 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:14.114688 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:14.114698 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:14.116574 (+ 1876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:14.119199 (+ 2625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:14.119210 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:14.119214 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:14.119580 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:14.119839 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:14.119840 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:14.119840 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:14.120007 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:14.120365 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:14.120372 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:14.120572 (+ 200us) client_negotiation.cc:769] Sending connection context
0925 15:20:14.120704 (+ 132us) client_negotiation.cc:240] Negotiation successful
0925 15:20:14.120924 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":74,"threads_started":1}
I20260925 15:20:14.122047 4680 heartbeater.cc:347] Connected to a master server at 127.3.211.254:42125
I20260925 15:20:14.122167 4680 heartbeater.cc:464] Registering TS with master...
I20260925 15:20:14.122416 4680 heartbeater.cc:511] Master 127.3.211.254:42125 requested a full tablet report, sending...
I20260925 15:20:14.123186 4489 ts_manager.cc:194] Registered new tserver with Master: 6067fe3a9c574c6dab76783fd1e2b56b (127.3.211.193:42085)
I20260925 15:20:14.124169 4489 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.193@KRBTEST.COM'} at 127.3.211.193:35257
WARNING: no policy specified for kudu/127.3.211.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab.
Entry for principal kudu/127.3.211.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab.
I20260925 15:20:14.143747 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.194:0
--local_ip_for_outbound_sockets=127.3.211.194
--webserver_interface=127.3.211.194
--webserver_port=0
--tserver_master_addrs=127.3.211.254:42125
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab
--principal=kudu/127.3.211.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:46835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:20:14.230897 4685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:14.231118 4685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:14.231143 4685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:14.232442 4685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:14.232501 4685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:14.232529 4685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.194
I20260925 15:20:14.234122 4685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:46835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab
--principal=kudu/127.3.211.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.211.194
--webserver_port=0
--tserver_master_addrs=127.3.211.254:42125
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.194
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:14.234416 4685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:14.234668 4685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:14.239982 4691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:14.239986 4693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:14.240015 4685 server_base.cc:1034] running on GCE node
W20260925 15:20:14.239980 4690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:14.240438 4685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:14.240763 4685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:14.241947 4685 hybrid_clock.cc:648] HybridClock initialized: now 1790349614241928 us; error 30 us; skew 500 ppm
Sep 25 15:20:14 dist-test-slave-4bk3 krb5kdc[4444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349614, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:14.244853 4685 init.cc:377] Logged in from keytab as kudu/127.3.211.194@KRBTEST.COM (short username kudu)
I20260925 15:20:14.245487 4685 webserver.cc:460] Webserver started at http://127.3.211.194:44769/ using document root <none> and password file <none>
I20260925 15:20:14.245806 4685 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:14.245883 4685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:14.246025 4685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:14.247143 4685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/ts-1/data/instance:
uuid: "8f90fbe5c1b34ec1b0ab80a02dbcfbd4"
format_stamp: "Formatted at 2026-09-25 15:20:14 on dist-test-slave-4bk3"
server_key: "79dc9cd945ac5dfcbeb3c203948b8b31"
server_key_iv: "ca7660b2dd8e88beb04d088d9c8f9206"
server_key_version: "encryptionkey@0"
I20260925 15:20:14.247532 4685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/ts-1/wal/instance:
uuid: "8f90fbe5c1b34ec1b0ab80a02dbcfbd4"
format_stamp: "Formatted at 2026-09-25 15:20:14 on dist-test-slave-4bk3"
server_key: "79dc9cd945ac5dfcbeb3c203948b8b31"
server_key_iv: "ca7660b2dd8e88beb04d088d9c8f9206"
server_key_version: "encryptionkey@0"
I20260925 15:20:14.249157 4685 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:20:14.250293 4700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:14.250577 4685 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:14.250721 4685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/ts-1/wal
uuid: "8f90fbe5c1b34ec1b0ab80a02dbcfbd4"
format_stamp: "Formatted at 2026-09-25 15:20:14 on dist-test-slave-4bk3"
server_key: "79dc9cd945ac5dfcbeb3c203948b8b31"
server_key_iv: "ca7660b2dd8e88beb04d088d9c8f9206"
server_key_version: "encryptionkey@0"
I20260925 15:20:14.250809 4685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:14.265076 4685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:14.267820 4685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:14.267949 4685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:14.268194 4685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:20:14.268543 4685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:20:14.268580 4685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:14.268607 4685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:20:14.268658 4685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:14.274688 4685 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.194:34387
I20260925 15:20:14.274791 4813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.194:34387 every 8 connection(s)
I20260925 15:20:14.275192 4685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
Sep 25 15:20:14 dist-test-slave-4bk3 krb5kdc[4444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349614, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.194@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:20:14.285138 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 4685
I20260925 15:20:14.285264 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/ts-1/wal/instance
I20260925 15:20:14.285423 3919 external_mini_cluster.cc:1428] Setting key 53f6b6f36f8677d69499e829bea1a11b
I20260925 15:20:14.285643 4528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:14.276107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:59569 (local address 127.3.211.254:42125)
0925 15:20:14.276232 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:14.276235 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:14.276439 (+ 204us) server_negotiation.cc:407] Connection header received
0925 15:20:14.277590 (+ 1151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:14.277591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:14.277603 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:14.277654 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:14.278329 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:14.278842 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:14.279885 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:14.280103 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:14.280200 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:14.282531 (+ 2331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:14.282533 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:14.282533 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:14.282542 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:14.283919 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:14.284516 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:14.284517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:14.284518 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:14.284587 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:14.284881 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:14.284882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:14.284883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:14.285024 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:14.285121 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:14.285445 (+ 324us) server_negotiation.cc:299] Negotiation successful
0925 15:20:14.285540 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 15:20:14.285866 4816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:14.276058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:42125 (local address 127.3.211.194:59569)
0925 15:20:14.276253 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:14.276301 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:14.277464 (+ 1163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:14.277729 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:14.277734 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:14.278022 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:14.278228 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:14.278232 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:14.278943 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:14.278944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:14.279780 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:14.279783 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:14.280214 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:14.280215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:14.280418 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:14.280966 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:14.280976 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:14.282417 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:14.284064 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:14.284073 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:14.284076 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:14.284416 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:14.284675 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:14.284676 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:14.284676 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:14.284799 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:14.285113 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:14.285120 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:14.285294 (+ 174us) client_negotiation.cc:769] Sending connection context
0925 15:20:14.285410 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 15:20:14.285601 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":86,"threads_started":1}
I20260925 15:20:14.286418 4814 heartbeater.cc:347] Connected to a master server at 127.3.211.254:42125
I20260925 15:20:14.286540 4814 heartbeater.cc:464] Registering TS with master...
I20260925 15:20:14.286741 4814 heartbeater.cc:511] Master 127.3.211.254:42125 requested a full tablet report, sending...
I20260925 15:20:14.287129 4489 ts_manager.cc:194] Registered new tserver with Master: 8f90fbe5c1b34ec1b0ab80a02dbcfbd4 (127.3.211.194:34387)
I20260925 15:20:14.287537 4489 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.194@KRBTEST.COM'} at 127.3.211.194:59569
WARNING: no policy specified for kudu/127.3.211.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab.
Entry for principal kudu/127.3.211.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab.
I20260925 15:20:14.312525 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.195:0
--local_ip_for_outbound_sockets=127.3.211.195
--webserver_interface=127.3.211.195
--webserver_port=0
--tserver_master_addrs=127.3.211.254:42125
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab
--principal=kudu/127.3.211.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:46835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:20:14.401137 4819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:14.401371 4819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:14.401398 4819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:14.402613 4819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:14.402659 4819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:14.402688 4819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.195
I20260925 15:20:14.404218 4819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:46835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab
--principal=kudu/127.3.211.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.211.195
--webserver_port=0
--tserver_master_addrs=127.3.211.254:42125
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.195
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:14.404525 4819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:14.404737 4819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:14.410104 4825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:14.410130 4827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:14.410188 4819 server_base.cc:1034] running on GCE node
W20260925 15:20:14.410102 4824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:14.410635 4819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:14.410881 4819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:14.412056 4819 hybrid_clock.cc:648] HybridClock initialized: now 1790349614412035 us; error 44 us; skew 500 ppm
Sep 25 15:20:14 dist-test-slave-4bk3 krb5kdc[4444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349614, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:14.414841 4819 init.cc:377] Logged in from keytab as kudu/127.3.211.195@KRBTEST.COM (short username kudu)
I20260925 15:20:14.415378 4819 webserver.cc:460] Webserver started at http://127.3.211.195:36337/ using document root <none> and password file <none>
I20260925 15:20:14.415634 4819 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:14.415705 4819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:14.415856 4819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:14.417023 4819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/ts-2/data/instance:
uuid: "99f4b1cbb44740ae8f210a8fc0388ada"
format_stamp: "Formatted at 2026-09-25 15:20:14 on dist-test-slave-4bk3"
server_key: "019c5f23d96c6cbd23dd75615a46f38f"
server_key_iv: "ff4b9b6cfc94ae19afc0b7667148996a"
server_key_version: "encryptionkey@0"
I20260925 15:20:14.417383 4819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/ts-2/wal/instance:
uuid: "99f4b1cbb44740ae8f210a8fc0388ada"
format_stamp: "Formatted at 2026-09-25 15:20:14 on dist-test-slave-4bk3"
server_key: "019c5f23d96c6cbd23dd75615a46f38f"
server_key_iv: "ff4b9b6cfc94ae19afc0b7667148996a"
server_key_version: "encryptionkey@0"
I20260925 15:20:14.418777 4819 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:20:14.419698 4834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:14.420274 4819 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:14.420498 4819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/ts-2/wal
uuid: "99f4b1cbb44740ae8f210a8fc0388ada"
format_stamp: "Formatted at 2026-09-25 15:20:14 on dist-test-slave-4bk3"
server_key: "019c5f23d96c6cbd23dd75615a46f38f"
server_key_iv: "ff4b9b6cfc94ae19afc0b7667148996a"
server_key_version: "encryptionkey@0"
I20260925 15:20:14.420607 4819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:14.434401 4819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:14.436914 4819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:14.437109 4819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:14.437408 4819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:20:14.437767 4819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:20:14.437824 4819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:14.437865 4819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:20:14.437888 4819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:14.445372 4819 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.195:40611
I20260925 15:20:14.445597 4947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.195:40611 every 8 connection(s)
I20260925 15:20:14.445856 4819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
I20260925 15:20:14.447923 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 4819
I20260925 15:20:14.448038 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790349610595785-3919-0/minicluster-data/ts-2/wal/instance
I20260925 15:20:14.448204 3919 external_mini_cluster.cc:1428] Setting key 2bb67509f346469709f75f4b706cd9a5
Sep 25 15:20:14 dist-test-slave-4bk3 krb5kdc[4444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349614, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.195@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:20:14.457015 4528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:14.446756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:35045 (local address 127.3.211.254:42125)
0925 15:20:14.446845 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:14.446847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:14.447059 (+ 212us) server_negotiation.cc:407] Connection header received
0925 15:20:14.448236 (+ 1177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:14.448255 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:14.448271 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:14.448411 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:14.449423 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:14.449950 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:14.451025 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:14.451222 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:14.451313 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:14.453863 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:14.453865 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:14.453865 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:14.453872 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:14.455464 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:14.455909 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:14.455911 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:14.455911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:14.455968 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:14.456321 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:14.456322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:14.456322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:14.456507 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:14.456567 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:14.456862 (+ 295us) server_negotiation.cc:299] Negotiation successful
0925 15:20:14.456941 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 15:20:14.457193 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:14.446667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:42125 (local address 127.3.211.195:35045)
0925 15:20:14.446947 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:14.446981 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:14.448104 (+ 1123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:14.448528 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:14.448532 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:14.448963 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:14.449323 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:14.449328 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:14.450062 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:14.450063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:14.450926 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:14.450929 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:14.451353 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:14.451354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:14.451488 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:14.452123 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:14.452140 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:14.453753 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:14.455562 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:14.455570 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:14.455572 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:14.455823 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:14.456100 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:14.456101 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:14.456102 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:14.456247 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:14.456571 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:14.456576 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:14.456711 (+ 135us) client_negotiation.cc:769] Sending connection context
0925 15:20:14.456806 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 15:20:14.456960 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":117,"threads_started":1}
I20260925 15:20:14.457726 4948 heartbeater.cc:347] Connected to a master server at 127.3.211.254:42125
I20260925 15:20:14.457849 4948 heartbeater.cc:464] Registering TS with master...
I20260925 15:20:14.458057 4948 heartbeater.cc:511] Master 127.3.211.254:42125 requested a full tablet report, sending...
I20260925 15:20:14.458657 4489 ts_manager.cc:194] Registered new tserver with Master: 99f4b1cbb44740ae8f210a8fc0388ada (127.3.211.195:40611)
I20260925 15:20:14.459223 4489 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.195@KRBTEST.COM'} at 127.3.211.195:35045
I20260925 15:20:14.459878 3919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:20:14.468297 4528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:14.461740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52532 (local address 127.3.211.254:42125)
0925 15:20:14.461815 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:14.461818 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:14.461878 (+ 60us) server_negotiation.cc:407] Connection header received
0925 15:20:14.461920 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:14.461921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:14.461930 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:14.461982 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:14.462569 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:14.463060 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:14.463832 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:14.464035 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:14.464101 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:14.465520 (+ 1419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:14.465521 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:14.465521 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:14.465528 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:14.466956 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:14.467344 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:14.467345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:14.467345 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:14.467413 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:14.467645 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:14.467646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:14.467646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:14.467818 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:14.467949 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:14.468098 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 15:20:14.468178 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 15:20:14.471567 4489 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52532:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:20:14.473520 4489 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:20:14.481173 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:14.476580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50576 (local address 127.3.211.195:40611)
0925 15:20:14.477015 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:14.477019 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:14.477029 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:20:14.477064 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:14.477066 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:14.477082 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:14.477157 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:20:14.477315 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:14.478420 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:14.479988 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:14.480754 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:14.480821 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:14.480882 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:14.481042 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 15:20:14.481086 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":92,"threads_started":1}
I20260925 15:20:14.481168 4959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:14.476489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.195:40611 (local address 127.0.0.1:50576)
0925 15:20:14.476833 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:14.476843 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:14.476893 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:14.477167 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:14.477168 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:14.477169 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:20:14.477233 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:14.477236 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:14.478570 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:14.478571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:14.479889 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:14.479891 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:14.480822 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:14.480823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:14.480908 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:14.480923 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 15:20:14.481027 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 15:20:14.481086 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":5,"thread_start_us":58,"threads_started":1}
I20260925 15:20:14.481751 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:14.476707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:42085 (local address 127.0.0.1:55166)
0925 15:20:14.476935 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:14.476944 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:14.476984 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:14.477493 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:14.477494 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:14.477496 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:20:14.477543 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:14.477565 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:14.478788 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:14.478789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:14.479904 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:14.479906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:14.481154 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:14.481155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:14.481220 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:14.481224 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:20:14.481638 (+ 414us) client_negotiation.cc:240] Negotiation successful
0925 15:20:14.481685 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":2,"thread_start_us":70,"threads_started":1}
I20260925 15:20:14.481823 4964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:14.476814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55166 (local address 127.3.211.193:42085)
0925 15:20:14.477333 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:14.477337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:14.477347 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:20:14.477384 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:14.477386 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:14.477400 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:14.477480 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:20:14.477682 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:14.478681 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:14.480412 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:14.481005 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:14.481074 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:14.481121 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:14.481660 (+ 539us) server_negotiation.cc:299] Negotiation successful
0925 15:20:14.481736 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":80,"threads_started":1}
I20260925 15:20:14.482048 4882 tablet_service.cc:1467] Processing CreateTablet for tablet 61298f99d3954e20908301741aa0ca1e (DEFAULT_TABLE table=test-table [id=e5f1bab89aaa4c10b39064a08ba0de56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:20:14.482443 4882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61298f99d3954e20908301741aa0ca1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:14.482686 4614 tablet_service.cc:1467] Processing CreateTablet for tablet 61298f99d3954e20908301741aa0ca1e (DEFAULT_TABLE table=test-table [id=e5f1bab89aaa4c10b39064a08ba0de56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:20:14.482969 4614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61298f99d3954e20908301741aa0ca1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:14.485992 4965 tablet_bootstrap.cc:492] T 61298f99d3954e20908301741aa0ca1e P 6067fe3a9c574c6dab76783fd1e2b56b: Bootstrap starting.
I20260925 15:20:14.486325 4966 tablet_bootstrap.cc:492] T 61298f99d3954e20908301741aa0ca1e P 99f4b1cbb44740ae8f210a8fc0388ada: Bootstrap starting.
I20260925 15:20:14.486922 4960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:14.476488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.194:34387 (local address 127.0.0.1:51888)
0925 15:20:14.476632 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:14.476674 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:14.476746 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:14.477135 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:14.477138 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:14.477140 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:20:14.477233 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:14.477236 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:14.479511 (+ 2275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:14.479512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:14.480672 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:14.480675 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:14.486565 (+ 5890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:14.486567 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:14.486674 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:14.486683 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 15:20:14.486773 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 15:20:14.486835 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":2,"thread_start_us":94,"threads_started":1}
I20260925 15:20:14.487071 4963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:14.476780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51888 (local address 127.3.211.194:34387)
0925 15:20:14.476930 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:14.476935 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:14.476946 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:20:14.476988 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:14.476991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:14.477007 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:14.477121 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:20:14.478561 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:14.479317 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:14.480765 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:14.486412 (+ 5647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:14.486566 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:14.486661 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:14.486869 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 15:20:14.486945 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":82,"threads_started":1}
I20260925 15:20:14.487130 4966 tablet_bootstrap.cc:654] T 61298f99d3954e20908301741aa0ca1e P 99f4b1cbb44740ae8f210a8fc0388ada: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:14.487147 4965 tablet_bootstrap.cc:654] T 61298f99d3954e20908301741aa0ca1e P 6067fe3a9c574c6dab76783fd1e2b56b: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:14.487736 4965 log.cc:824] T 61298f99d3954e20908301741aa0ca1e P 6067fe3a9c574c6dab76783fd1e2b56b: Log is configured to *not* fsync() on all Append() calls
I20260925 15:20:14.487766 4966 log.cc:824] T 61298f99d3954e20908301741aa0ca1e P 99f4b1cbb44740ae8f210a8fc0388ada: Log is configured to *not* fsync() on all Append() calls
I20260925 15:20:14.488253 4748 tablet_service.cc:1467] Processing CreateTablet for tablet 61298f99d3954e20908301741aa0ca1e (DEFAULT_TABLE table=test-table [id=e5f1bab89aaa4c10b39064a08ba0de56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:20:14.488673 4748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61298f99d3954e20908301741aa0ca1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:14.488966 4966 tablet_bootstrap.cc:492] T 61298f99d3954e20908301741aa0ca1e P 99f4b1cbb44740ae8f210a8fc0388ada: No bootstrap required, opened a new log
I20260925 15:20:14.488976 4965 tablet_bootstrap.cc:492] T 61298f99d3954e20908301741aa0ca1e P 6067fe3a9c574c6dab76783fd1e2b56b: No bootstrap required, opened a new log
I20260925 15:20:14.489048 4966 ts_tablet_manager.cc:1397] T 61298f99d3954e20908301741aa0ca1e P 99f4b1cbb44740ae8f210a8fc0388ada: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 15:20:14.489070 4965 ts_tablet_manager.cc:1397] T 61298f99d3954e20908301741aa0ca1e P 6067fe3a9c574c6dab76783fd1e2b56b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 15:20:14.490506 4965 raft_consensus.cc:348] T 61298f99d3954e20908301741aa0ca1e P 6067fe3a9c574c6dab76783fd1e2b56b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90fbe5c1b34ec1b0ab80a02dbcfbd4" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 34387 } } peers { permanent_uuid: "99f4b1cbb44740ae8f210a8fc0388ada" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 40611 } } peers { permanent_uuid: "6067fe3a9c574c6dab76783fd1e2b56b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 42085 } }
I20260925 15:20:14.490701 4965 raft_consensus.cc:374] T 61298f99d3954e20908301741aa0ca1e P 6067fe3a9c574c6dab76783fd1e2b56b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:14.490746 4965 raft_consensus.cc:729] T 61298f99d3954e20908301741aa0ca1e P 6067fe3a9c574c6dab76783fd1e2b56b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6067fe3a9c574c6dab76783fd1e2b56b, State: Initialized, Role: FOLLOWER
I20260925 15:20:14.490891 4965 consensus_queue.cc:260] T 61298f99d3954e20908301741aa0ca1e P 6067fe3a9c574c6dab76783fd1e2b56b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90fbe5c1b34ec1b0ab80a02dbcfbd4" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 34387 } } peers { permanent_uuid: "99f4b1cbb44740ae8f210a8fc0388ada" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 40611 } } peers { permanent_uuid: "6067fe3a9c574c6dab76783fd1e2b56b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 42085 } }
I20260925 15:20:14.491222 4680 heartbeater.cc:503] Master 127.3.211.254:42125 was elected leader, sending a full tablet report...
I20260925 15:20:14.491170 4966 raft_consensus.cc:348] T 61298f99d3954e20908301741aa0ca1e P 99f4b1cbb44740ae8f210a8fc0388ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90fbe5c1b34ec1b0ab80a02dbcfbd4" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 34387 } } peers { permanent_uuid: "99f4b1cbb44740ae8f210a8fc0388ada" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 40611 } } peers { permanent_uuid: "6067fe3a9c574c6dab76783fd1e2b56b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 42085 } }
I20260925 15:20:14.491339 4966 raft_consensus.cc:374] T 61298f99d3954e20908301741aa0ca1e P 99f4b1cbb44740ae8f210a8fc0388ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:14.491382 4966 raft_consensus.cc:729] T 61298f99d3954e20908301741aa0ca1e P 99f4b1cbb44740ae8f210a8fc0388ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99f4b1cbb44740ae8f210a8fc0388ada, State: Initialized, Role: FOLLOWER
I20260925 15:20:14.491539 4966 consensus_queue.cc:260] T 61298f99d3954e20908301741aa0ca1e P 99f4b1cbb44740ae8f210a8fc0388ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90fbe5c1b34ec1b0ab80a02dbcfbd4" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 34387 } } peers { permanent_uuid: "99f4b1cbb44740ae8f210a8fc0388ada" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 40611 } } peers { permanent_uuid: "6067fe3a9c574c6dab76783fd1e2b56b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 42085 } }
I20260925 15:20:14.491590 4965 ts_tablet_manager.cc:1428] T 61298f99d3954e20908301741aa0ca1e P 6067fe3a9c574c6dab76783fd1e2b56b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 15:20:14.491822 4966 ts_tablet_manager.cc:1428] T 61298f99d3954e20908301741aa0ca1e P 99f4b1cbb44740ae8f210a8fc0388ada: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:20:14.491846 4948 heartbeater.cc:503] Master 127.3.211.254:42125 was elected leader, sending a full tablet report...
I20260925 15:20:14.492151 4971 tablet_bootstrap.cc:492] T 61298f99d3954e20908301741aa0ca1e P 8f90fbe5c1b34ec1b0ab80a02dbcfbd4: Bootstrap starting.
I20260925 15:20:14.493083 4971 tablet_bootstrap.cc:654] T 61298f99d3954e20908301741aa0ca1e P 8f90fbe5c1b34ec1b0ab80a02dbcfbd4: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:14.493549 4971 log.cc:824] T 61298f99d3954e20908301741aa0ca1e P 8f90fbe5c1b34ec1b0ab80a02dbcfbd4: Log is configured to *not* fsync() on all Append() calls
I20260925 15:20:14.494268 4971 tablet_bootstrap.cc:492] T 61298f99d3954e20908301741aa0ca1e P 8f90fbe5c1b34ec1b0ab80a02dbcfbd4: No bootstrap required, opened a new log
I20260925 15:20:14.494338 4971 ts_tablet_manager.cc:1397] T 61298f99d3954e20908301741aa0ca1e P 8f90fbe5c1b34ec1b0ab80a02dbcfbd4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:20:14.495765 4971 raft_consensus.cc:348] T 61298f99d3954e20908301741aa0ca1e P 8f90fbe5c1b34ec1b0ab80a02dbcfbd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90fbe5c1b34ec1b0ab80a02dbcfbd4" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 34387 } } peers { permanent_uuid: "99f4b1cbb44740ae8f210a8fc0388ada" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 40611 } } peers { permanent_uuid: "6067fe3a9c574c6dab76783fd1e2b56b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 42085 } }
I20260925 15:20:14.495934 4971 raft_consensus.cc:374] T 61298f99d3954e20908301741aa0ca1e P 8f90fbe5c1b34ec1b0ab80a02dbcfbd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:14.495958 4971 raft_consensus.cc:729] T 61298f99d3954e20908301741aa0ca1e P 8f90fbe5c1b34ec1b0ab80a02dbcfbd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f90fbe5c1b34ec1b0ab80a02dbcfbd4, State: Initialized, Role: FOLLOWER
I20260925 15:20:14.496124 4971 consensus_queue.cc:260] T 61298f99d3954e20908301741aa0ca1e P 8f90fbe5c1b34ec1b0ab80a02dbcfbd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90fbe5c1b34ec1b0ab80a02dbcfbd4" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 34387 } } peers { permanent_uuid: "99f4b1cbb44740ae8f210a8fc0388ada" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 40611 } } peers { permanent_uuid: "6067fe3a9c574c6dab76783fd1e2b56b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 42085 } }
I20260925 15:20:14.496477 4971 ts_tablet_manager.cc:1428] T 61298f99d3954e20908301741aa0ca1e P 8f90fbe5c1b34ec1b0ab80a02dbcfbd4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:20:14.496559 4814 heartbeater.cc:503] Master 127.3.211.254:42125 was elected leader, sending a full tablet report...
W20260925 15:20:14.525868 4815 tablet.cc:2369] T 61298f99d3954e20908301741aa0ca1e P 8f90fbe5c1b34ec1b0ab80a02dbcfbd4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:20:14.551347 4969 raft_consensus.cc:482] T 61298f99d3954e20908301741aa0ca1e P 6067fe3a9c574c6dab76783fd1e2b56b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:20:14.551548 4969 raft_consensus.cc:504] T 61298f99d3954e20908301741aa0ca1e P 6067fe3a9c574c6dab76783fd1e2b56b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90fbe5c1b34ec1b0ab80a02dbcfbd4" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 34387 } } peers { permanent_uuid: "99f4b1cbb44740ae8f210a8fc0388ada" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 40611 } } peers { permanent_uuid: "6067fe3a9c574c6dab76783fd1e2b56b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 42085 } }
I20260925 15:20:14.552057 4969 leader_election.cc:288] T 61298f99d3954e20908301741aa0ca1e P 6067fe3a9c574c6dab76783fd1e2b56b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f90fbe5c1b34ec1b0ab80a02dbcfbd4 (127.3.211.194:34387), 99f4b1cbb44740ae8f210a8fc0388ada (127.3.211.195:40611)
I20260925 15:20:14.555661 4682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:14.552236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.194:34387 (local address 127.3.211.193:44557)
0925 15:20:14.552308 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:14.552323 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:14.552394 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:14.552881 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:14.552883 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:14.552885 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:20:14.552971 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:14.552973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:14.553836 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:14.553837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:14.554764 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:14.554766 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:14.555368 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:14.555369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:14.555448 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:14.555451 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:20:14.555529 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 15:20:14.555573 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 15:20:14.555652 4963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:14.552364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:44557 (local address 127.3.211.194:34387)
0925 15:20:14.552414 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:14.552416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:14.552425 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:20:14.552473 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:14.552474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:14.552489 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:14.552574 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:20:14.553078 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:14.553743 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:14.554854 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:14.555275 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:14.555372 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:14.555383 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:14.555537 (+ 154us) server_negotiation.cc:299] Negotiation successful
0925 15:20:14.555577 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 15:20:14.555931 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:14.552356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:41065 (local address 127.3.211.195:40611)
0925 15:20:14.552407 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:14.552409 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:14.552633 (+ 224us) server_negotiation.cc:407] Connection header received
0925 15:20:14.552691 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:14.552692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:14.552705 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:14.552807 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:20:14.553327 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:14.553873 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:14.555090 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:14.555506 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:14.555611 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:14.555620 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:14.555804 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 15:20:14.555865 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 15:20:14.556136 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:14.552268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.195:40611 (local address 127.3.211.193:41065)
0925 15:20:14.552491 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:14.552502 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:14.552569 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:14.552946 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:14.552947 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:14.552949 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:20:14.553083 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:14.553086 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:14.554073 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:14.554075 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:14.554988 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:14.554990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:14.555616 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:14.555617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:14.555688 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:14.555697 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 15:20:14.556018 (+ 321us) client_negotiation.cc:240] Negotiation successful
0925 15:20:14.556059 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":140,"threads_started":1}
I20260925 15:20:14.556154 4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61298f99d3954e20908301741aa0ca1e" candidate_uuid: "6067fe3a9c574c6dab76783fd1e2b56b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90fbe5c1b34ec1b0ab80a02dbcfbd4" is_pre_election: true
I20260925 15:20:14.556419 4768 raft_consensus.cc:2454] T 61298f99d3954e20908301741aa0ca1e P 8f90fbe5c1b34ec1b0ab80a02dbcfbd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6067fe3a9c574c6dab76783fd1e2b56b in term 0.
I20260925 15:20:14.556793 4568 leader_election.cc:302] T 61298f99d3954e20908301741aa0ca1e P 6067fe3a9c574c6dab76783fd1e2b56b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6067fe3a9c574c6dab76783fd1e2b56b, 8f90fbe5c1b34ec1b0ab80a02dbcfbd4; no voters:
I20260925 15:20:14.556862 4902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61298f99d3954e20908301741aa0ca1e" candidate_uuid: "6067fe3a9c574c6dab76783fd1e2b56b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99f4b1cbb44740ae8f210a8fc0388ada" is_pre_election: true
I20260925 15:20:14.556993 4902 raft_consensus.cc:2454] T 61298f99d3954e20908301741aa0ca1e P 99f4b1cbb44740ae8f210a8fc0388ada [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6067fe3a9c574c6dab76783fd1e2b56b in term 0.
I20260925 15:20:14.557114 4969 raft_consensus.cc:2781] T 61298f99d3954e20908301741aa0ca1e P 6067fe3a9c574c6dab76783fd1e2b56b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:20:14.557199 4969 raft_consensus.cc:482] T 61298f99d3954e20908301741aa0ca1e P 6067fe3a9c574c6dab76783fd1e2b56b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:20:14.557237 4969 raft_consensus.cc:3037] T 61298f99d3954e20908301741aa0ca1e P 6067fe3a9c574c6dab76783fd1e2b56b [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:14.557951 4969 raft_consensus.cc:504] T 61298f99d3954e20908301741aa0ca1e P 6067fe3a9c574c6dab76783fd1e2b56b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90fbe5c1b34ec1b0ab80a02dbcfbd4" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 34387 } } peers { permanent_uuid: "99f4b1cbb44740ae8f210a8fc0388ada" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 40611 } } peers { permanent_uuid: "6067fe3a9c574c6dab76783fd1e2b56b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 42085 } }
I20260925 15:20:14.558117 4969 leader_election.cc:288] T 61298f99d3954e20908301741aa0ca1e P 6067fe3a9c574c6dab76783fd1e2b56b [CANDIDATE]: Term 1 election: Requested vote from peers 8f90fbe5c1b34ec1b0ab80a02dbcfbd4 (127.3.211.194:34387), 99f4b1cbb44740ae8f210a8fc0388ada (127.3.211.195:40611)
I20260925 15:20:14.558328 4902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61298f99d3954e20908301741aa0ca1e" candidate_uuid: "6067fe3a9c574c6dab76783fd1e2b56b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99f4b1cbb44740ae8f210a8fc0388ada"
I20260925 15:20:14.558338 4768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61298f99d3954e20908301741aa0ca1e" candidate_uuid: "6067fe3a9c574c6dab76783fd1e2b56b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f90fbe5c1b34ec1b0ab80a02dbcfbd4"
I20260925 15:20:14.558434 4768 raft_consensus.cc:3037] T 61298f99d3954e20908301741aa0ca1e P 8f90fbe5c1b34ec1b0ab80a02dbcfbd4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:14.558439 4902 raft_consensus.cc:3037] T 61298f99d3954e20908301741aa0ca1e P 99f4b1cbb44740ae8f210a8fc0388ada [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:14.559214 4768 raft_consensus.cc:2454] T 61298f99d3954e20908301741aa0ca1e P 8f90fbe5c1b34ec1b0ab80a02dbcfbd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6067fe3a9c574c6dab76783fd1e2b56b in term 1.
I20260925 15:20:14.559221 4902 raft_consensus.cc:2454] T 61298f99d3954e20908301741aa0ca1e P 99f4b1cbb44740ae8f210a8fc0388ada [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6067fe3a9c574c6dab76783fd1e2b56b in term 1.
I20260925 15:20:14.559396 4568 leader_election.cc:302] T 61298f99d3954e20908301741aa0ca1e P 6067fe3a9c574c6dab76783fd1e2b56b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6067fe3a9c574c6dab76783fd1e2b56b, 8f90fbe5c1b34ec1b0ab80a02dbcfbd4; no voters:
I20260925 15:20:14.559538 4969 raft_consensus.cc:2781] T 61298f99d3954e20908301741aa0ca1e P 6067fe3a9c574c6dab76783fd1e2b56b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:20:14.559692 4969 raft_consensus.cc:686] T 61298f99d3954e20908301741aa0ca1e P 6067fe3a9c574c6dab76783fd1e2b56b [term 1 LEADER]: Becoming Leader. State: Replica: 6067fe3a9c574c6dab76783fd1e2b56b, State: Running, Role: LEADER
I20260925 15:20:14.559800 4969 consensus_queue.cc:237] T 61298f99d3954e20908301741aa0ca1e P 6067fe3a9c574c6dab76783fd1e2b56b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90fbe5c1b34ec1b0ab80a02dbcfbd4" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 34387 } } peers { permanent_uuid: "99f4b1cbb44740ae8f210a8fc0388ada" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 40611 } } peers { permanent_uuid: "6067fe3a9c574c6dab76783fd1e2b56b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 42085 } }
I20260925 15:20:14.560835 4489 catalog_manager.cc:5495] T 61298f99d3954e20908301741aa0ca1e P 6067fe3a9c574c6dab76783fd1e2b56b reported cstate change: term changed from 0 to 1, leader changed from <none> to 6067fe3a9c574c6dab76783fd1e2b56b (127.3.211.193). New cstate: current_term: 1 leader_uuid: "6067fe3a9c574c6dab76783fd1e2b56b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f90fbe5c1b34ec1b0ab80a02dbcfbd4" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 34387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99f4b1cbb44740ae8f210a8fc0388ada" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 40611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6067fe3a9c574c6dab76783fd1e2b56b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 42085 } health_report { overall_health: HEALTHY } } }
Sep 25 15:20:14 dist-test-slave-4bk3 krb5kdc[4444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349613, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.211.193@KRBTEST.COM
I20260925 15:20:14.581925 4964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:14.571986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55180 (local address 127.3.211.193:42085)
0925 15:20:14.572070 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:14.572072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:14.572122 (+ 50us) server_negotiation.cc:407] Connection header received
0925 15:20:14.572235 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:14.572236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:14.572258 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:14.572532 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:14.573176 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:14.573654 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:14.574427 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:14.574630 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:14.574709 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:14.578618 (+ 3909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:14.578639 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:14.578639 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:14.578648 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:14.580648 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:14.581064 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:14.581068 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:14.581068 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:14.581129 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:14.581355 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:14.581356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:14.581357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:14.581509 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:14.581692 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:14.581731 (+ 39us) server_negotiation.cc:299] Negotiation successful
0925 15:20:14.581815 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Sep 25 15:20:14 dist-test-slave-4bk3 krb5kdc[4444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349614, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 15:20:14 dist-test-slave-4bk3 krb5kdc[4444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349614, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.211.193@KRBTEST.COM
I20260925 15:20:14.607821 4964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:14.599342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55186 (local address 127.3.211.193:42085)
0925 15:20:14.599431 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:14.599432 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:14.599469 (+ 37us) server_negotiation.cc:407] Connection header received
0925 15:20:14.599562 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:14.599563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:14.599573 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:14.599632 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:14.600301 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:14.600905 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:14.601693 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:14.601895 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:14.601960 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:14.604899 (+ 2939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:14.604901 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:14.604901 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:14.604909 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:14.606670 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:14.607048 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:14.607049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:14.607049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:14.607105 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:14.607315 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:14.607316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:14.607317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:14.607440 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:14.607532 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:14.607679 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 15:20:14.607750 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260925 15:20:14.610000 4681 tablet.cc:2369] T 61298f99d3954e20908301741aa0ca1e P 6067fe3a9c574c6dab76783fd1e2b56b: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 15:20:14 dist-test-slave-4bk3 krb5kdc[4444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 15:20:14 dist-test-slave-4bk3 krb5kdc[4444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.211.193@KRBTEST.COM
I20260925 15:20:14.633211 4964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:14.625017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55194 (local address 127.3.211.193:42085)
0925 15:20:14.625092 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:14.625095 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:14.625197 (+ 102us) server_negotiation.cc:407] Connection header received
0925 15:20:14.625298 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:14.625299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:14.625310 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:14.625382 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:14.625975 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:14.626472 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:14.627221 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:14.627419 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:14.627511 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:14.630143 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:14.630145 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:14.630145 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:14.630154 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:14.631694 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:14.632091 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:14.632092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:14.632092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:14.632145 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:14.632527 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:14.632528 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:14.632528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:14.632652 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:14.632745 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:14.632959 (+ 214us) server_negotiation.cc:299] Negotiation successful
0925 15:20:14.633078 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260925 15:20:14.633776 4594 tablet_service.cc:2908] Rejecting scan request for tablet 61298f99d3954e20908301741aa0ca1e: Uninitialized: clean time has not yet been initialized
I20260925 15:20:14.634829 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 4551
I20260925 15:20:14.639990 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 4685
I20260925 15:20:14.645063 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 4819
I20260925 15:20:14.649813 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 4457
2026-09-25T15:20:14Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2099 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:20:14 dist-test-slave-4bk3 krb5kdc[4989](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:20:14 dist-test-slave-4bk3 krb5kdc[4989](info): set up 2 sockets
Sep 25 15:20:14 dist-test-slave-4bk3 krb5kdc[4989](info): commencing operation
krb5kdc: starting...
W20260925 15:20:15.700080 3919 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Sep 25 15:20:15 dist-test-slave-4bk3 krb5kdc[4989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:20:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:20:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.211.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
I20260925 15:20:15.841802 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:40471
--webserver_interface=127.3.211.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:36825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.211.254:40471
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:20:15.925839 5002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:15.926082 5002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:15.926108 5002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:15.927318 5002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:20:15.927358 5002 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 15:20:15.927373 5002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:15.927386 5002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:15.927399 5002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:20:15.927412 5002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:20:15.929018 5002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:36825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.211.254:40471
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:40471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.211.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:15.929245 5002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:15.929436 5002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:15.934717 5007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:15.934774 5002 server_base.cc:1034] running on GCE node
W20260925 15:20:15.934731 5008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:15.934727 5010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:15.935256 5002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:15.935560 5002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:15.936789 5002 hybrid_clock.cc:648] HybridClock initialized: now 1790349615936749 us; error 51 us; skew 500 ppm
Sep 25 15:20:15 dist-test-slave-4bk3 krb5kdc[4989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349615, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:15.939409 5002 init.cc:377] Logged in from keytab as kudu/127.3.211.254@KRBTEST.COM (short username kudu)
I20260925 15:20:15.940057 5002 webserver.cc:460] Webserver started at http://127.3.211.254:34303/ using document root <none> and password file <none>
I20260925 15:20:15.940277 5002 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:15.940320 5002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:15.940498 5002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:15.941571 5002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/master-0/data/instance:
uuid: "980593882e994988bb22962652e15a9d"
format_stamp: "Formatted at 2026-09-25 15:20:15 on dist-test-slave-4bk3"
server_key: "4e5b3ccd93d78222c19b9f0fb52b17c0"
server_key_iv: "808776edb67a0123b0a3d01689b720ef"
server_key_version: "encryptionkey@0"
I20260925 15:20:15.941884 5002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/master-0/wal/instance:
uuid: "980593882e994988bb22962652e15a9d"
format_stamp: "Formatted at 2026-09-25 15:20:15 on dist-test-slave-4bk3"
server_key: "4e5b3ccd93d78222c19b9f0fb52b17c0"
server_key_iv: "808776edb67a0123b0a3d01689b720ef"
server_key_version: "encryptionkey@0"
I20260925 15:20:15.943435 5002 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:20:15.944325 5017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:15.944610 5002 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:20:15.944729 5002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/master-0/wal
uuid: "980593882e994988bb22962652e15a9d"
format_stamp: "Formatted at 2026-09-25 15:20:15 on dist-test-slave-4bk3"
server_key: "4e5b3ccd93d78222c19b9f0fb52b17c0"
server_key_iv: "808776edb67a0123b0a3d01689b720ef"
server_key_version: "encryptionkey@0"
I20260925 15:20:15.944809 5002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:15.954393 5002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:15.956841 5002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:15.957849 5002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:15.964607 5002 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.254:40471
I20260925 15:20:15.964803 5079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.254:40471 every 8 connection(s)
I20260925 15:20:15.965130 5002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
I20260925 15:20:15.965958 5080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:15.967391 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 5002
I20260925 15:20:15.967514 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/master-0/wal/instance
I20260925 15:20:15.967701 3919 external_mini_cluster.cc:1428] Setting key 647116e7b9fda808ebb1b5259f013dea
I20260925 15:20:15.969107 5080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 980593882e994988bb22962652e15a9d: Bootstrap starting.
I20260925 15:20:15.969915 5080 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 980593882e994988bb22962652e15a9d: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:15.970257 5080 log.cc:824] T 00000000000000000000000000000000 P 980593882e994988bb22962652e15a9d: Log is configured to *not* fsync() on all Append() calls
I20260925 15:20:15.971087 5080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 980593882e994988bb22962652e15a9d: No bootstrap required, opened a new log
I20260925 15:20:15.974224 5080 raft_consensus.cc:348] T 00000000000000000000000000000000 P 980593882e994988bb22962652e15a9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "980593882e994988bb22962652e15a9d" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 40471 } }
I20260925 15:20:15.974464 5080 raft_consensus.cc:374] T 00000000000000000000000000000000 P 980593882e994988bb22962652e15a9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:15.974510 5080 raft_consensus.cc:729] T 00000000000000000000000000000000 P 980593882e994988bb22962652e15a9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 980593882e994988bb22962652e15a9d, State: Initialized, Role: FOLLOWER
I20260925 15:20:15.974673 5080 consensus_queue.cc:260] T 00000000000000000000000000000000 P 980593882e994988bb22962652e15a9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "980593882e994988bb22962652e15a9d" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 40471 } }
I20260925 15:20:15.974776 5080 raft_consensus.cc:388] T 00000000000000000000000000000000 P 980593882e994988bb22962652e15a9d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:20:15.974809 5080 raft_consensus.cc:482] T 00000000000000000000000000000000 P 980593882e994988bb22962652e15a9d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:20:15.974850 5080 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 980593882e994988bb22962652e15a9d [term 0 FOLLOWER]: Advancing to term 1
Sep 25 15:20:15 dist-test-slave-4bk3 krb5kdc[4989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:20:15.975704 5080 raft_consensus.cc:504] T 00000000000000000000000000000000 P 980593882e994988bb22962652e15a9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "980593882e994988bb22962652e15a9d" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 40471 } }
I20260925 15:20:15.975826 5080 leader_election.cc:302] T 00000000000000000000000000000000 P 980593882e994988bb22962652e15a9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 980593882e994988bb22962652e15a9d; no voters:
I20260925 15:20:15.976096 5080 leader_election.cc:288] T 00000000000000000000000000000000 P 980593882e994988bb22962652e15a9d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:20:15.976128 5085 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 980593882e994988bb22962652e15a9d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:20:15.976289 5085 raft_consensus.cc:686] T 00000000000000000000000000000000 P 980593882e994988bb22962652e15a9d [term 1 LEADER]: Becoming Leader. State: Replica: 980593882e994988bb22962652e15a9d, State: Running, Role: LEADER
I20260925 15:20:15.976418 5085 consensus_queue.cc:237] T 00000000000000000000000000000000 P 980593882e994988bb22962652e15a9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "980593882e994988bb22962652e15a9d" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 40471 } }
I20260925 15:20:15.976451 5080 sys_catalog.cc:564] T 00000000000000000000000000000000 P 980593882e994988bb22962652e15a9d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:20:15.976912 5086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 980593882e994988bb22962652e15a9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "980593882e994988bb22962652e15a9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "980593882e994988bb22962652e15a9d" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 40471 } } }
I20260925 15:20:15.977053 5086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 980593882e994988bb22962652e15a9d [sys.catalog]: This master's current role is: LEADER
I20260925 15:20:15.978279 5087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 980593882e994988bb22962652e15a9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 980593882e994988bb22962652e15a9d. Latest consensus state: current_term: 1 leader_uuid: "980593882e994988bb22962652e15a9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "980593882e994988bb22962652e15a9d" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 40471 } } }
I20260925 15:20:15.978364 5087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 980593882e994988bb22962652e15a9d [sys.catalog]: This master's current role is: LEADER
W20260925 15:20:15.978471 5100 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 980593882e994988bb22962652e15a9d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:20:15.978749 5100 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:20:15.978813 5101 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:20:15.978988 5101 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:20:15.979751 5083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:15.968410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53544 (local address 127.3.211.254:40471)
0925 15:20:15.968580 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:15.968591 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:15.968627 (+ 36us) server_negotiation.cc:407] Connection header received
0925 15:20:15.969199 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:15.969233 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:15.969309 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:15.969497 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:15.970155 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:15.971161 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:15.972263 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:15.972791 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:15.972893 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:15.975790 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:15.975794 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:15.975794 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:15.975833 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:15.977940 (+ 2107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:15.978442 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:15.978446 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:15.978446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:15.978559 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:15.978791 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:15.978792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:15.978792 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:15.979103 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:15.979260 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:15.979387 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 15:20:15.979534 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":81,"threads_started":1}
I20260925 15:20:15.980755 5101 catalog_manager.cc:1347] Generated new cluster ID: 31f76777a2bb4e92906c55a58db7debf
I20260925 15:20:15.980831 5101 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:20:15.995288 5101 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:20:15.996037 5101 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:20:16.000726 5101 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 980593882e994988bb22962652e15a9d: Generated new TSK 0
I20260925 15:20:16.000958 5101 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.211.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab.
Entry for principal kudu/127.3.211.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab.
I20260925 15:20:16.040095 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.193:0
--local_ip_for_outbound_sockets=127.3.211.193
--webserver_interface=127.3.211.193
--webserver_port=0
--tserver_master_addrs=127.3.211.254:40471
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab
--principal=kudu/127.3.211.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:36825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:20:16.147497 5106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:16.147789 5106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:16.147832 5106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:16.149408 5106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:16.149504 5106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:16.149544 5106 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:20:16.149585 5106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.193
I20260925 15:20:16.151392 5106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:36825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab
--principal=kudu/127.3.211.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.211.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.211.254:40471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.193
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:16.151722 5106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:16.151995 5106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:16.158495 5114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:16.158496 5112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:16.158762 5106 server_base.cc:1034] running on GCE node
W20260925 15:20:16.158524 5111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:16.159022 5106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:16.159308 5106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:16.160450 5106 hybrid_clock.cc:648] HybridClock initialized: now 1790349616160433 us; error 44 us; skew 500 ppm
Sep 25 15:20:16 dist-test-slave-4bk3 krb5kdc[4989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349616, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:16.163558 5106 init.cc:377] Logged in from keytab as kudu/127.3.211.193@KRBTEST.COM (short username kudu)
I20260925 15:20:16.164489 5106 webserver.cc:460] Webserver started at http://127.3.211.193:38721/ using document root <none> and password file <none>
I20260925 15:20:16.164980 5106 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:16.165130 5106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:16.165328 5106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:16.166540 5106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/ts-0/data/instance:
uuid: "aecdd2c927464303a8a661264435dc61"
format_stamp: "Formatted at 2026-09-25 15:20:16 on dist-test-slave-4bk3"
server_key: "41307ede8c229c3164dc526631126069"
server_key_iv: "9fc5e7572840478dd84aed0c67e74a26"
server_key_version: "encryptionkey@0"
I20260925 15:20:16.166968 5106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/ts-0/wal/instance:
uuid: "aecdd2c927464303a8a661264435dc61"
format_stamp: "Formatted at 2026-09-25 15:20:16 on dist-test-slave-4bk3"
server_key: "41307ede8c229c3164dc526631126069"
server_key_iv: "9fc5e7572840478dd84aed0c67e74a26"
server_key_version: "encryptionkey@0"
I20260925 15:20:16.168578 5106 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260925 15:20:16.169453 5121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:16.169697 5106 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:16.169811 5106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/ts-0/wal
uuid: "aecdd2c927464303a8a661264435dc61"
format_stamp: "Formatted at 2026-09-25 15:20:16 on dist-test-slave-4bk3"
server_key: "41307ede8c229c3164dc526631126069"
server_key_iv: "9fc5e7572840478dd84aed0c67e74a26"
server_key_version: "encryptionkey@0"
I20260925 15:20:16.169898 5106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:16.191529 5106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:16.193884 5106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:16.194074 5106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:16.194631 5106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:20:16.194703 5106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:16.194772 5106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:20:16.194819 5106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 15:20:16 dist-test-slave-4bk3 krb5kdc[4989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349616, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.193@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:20:16.204800 5106 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.193:35851
I20260925 15:20:16.204888 5236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.193:35851 every 8 connection(s)
I20260925 15:20:16.205235 5106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:20:16.207508 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 5106
I20260925 15:20:16.207674 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/ts-0/wal/instance
I20260925 15:20:16.207882 3919 external_mini_cluster.cc:1428] Setting key 6b1a54f4a608b61b4ef6784c1b384a43
I20260925 15:20:16.209125 5083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:16.195281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:49725 (local address 127.3.211.254:40471)
0925 15:20:16.195336 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:16.195338 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:16.195613 (+ 275us) server_negotiation.cc:407] Connection header received
0925 15:20:16.196226 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:16.196228 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:16.196244 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:16.196323 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:16.197292 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.198095 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.201286 (+ 3191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.201603 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.201720 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:16.205323 (+ 3603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:16.205326 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:16.205326 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:16.205335 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:16.206863 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:16.207890 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:16.207891 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:16.207892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:16.207992 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:16.208311 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:16.208312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:16.208312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:16.208470 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:16.208556 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:16.208911 (+ 355us) server_negotiation.cc:299] Negotiation successful
0925 15:20:16.209034 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 15:20:16.209331 5130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:16.195295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:40471 (local address 127.3.211.193:49725)
0925 15:20:16.195482 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:16.195518 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:16.196070 (+ 552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:16.196455 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:16.196461 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:16.196885 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:16.197192 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:16.197199 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.198220 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.198222 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:16.201088 (+ 2866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:16.201097 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.201751 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.201753 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:16.201925 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:16.203242 (+ 1317us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:16.203253 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:16.205182 (+ 1929us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:16.207056 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:16.207071 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:16.207075 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:16.207780 (+ 705us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:16.208072 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:16.208073 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:16.208074 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:16.208235 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:16.208547 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:16.208555 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:16.208757 (+ 202us) client_negotiation.cc:769] Sending connection context
0925 15:20:16.208872 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 15:20:16.209062 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":101,"threads_started":1}
I20260925 15:20:16.209822 5237 heartbeater.cc:347] Connected to a master server at 127.3.211.254:40471
I20260925 15:20:16.209946 5237 heartbeater.cc:464] Registering TS with master...
I20260925 15:20:16.210156 5237 heartbeater.cc:511] Master 127.3.211.254:40471 requested a full tablet report, sending...
I20260925 15:20:16.210738 5033 ts_manager.cc:194] Registered new tserver with Master: aecdd2c927464303a8a661264435dc61 (127.3.211.193:35851)
I20260925 15:20:16.211593 5033 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.193@KRBTEST.COM'} at 127.3.211.193:49725
WARNING: no policy specified for kudu/127.3.211.194@KRBTEST.COM; defaulting to no policy
I20260925 15:20:16.218572 5244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:16.210961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:40471 (local address 127.3.211.193:47071)
0925 15:20:16.211111 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:16.211122 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:16.211184 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:16.211323 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:16.211324 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:16.211585 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:16.211665 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:16.211667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.212631 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.212632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:16.213577 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:16.213580 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.213968 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.213969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:16.214055 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:16.214565 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:16.214572 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:16.215047 (+ 475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:16.217251 (+ 2204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:16.217253 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:16.217254 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:16.217522 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:16.217825 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:16.217826 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:16.217826 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:16.217904 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:16.218260 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:16.218261 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:16.218308 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 15:20:16.218369 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 15:20:16.218461 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":80,"threads_started":1}
I20260925 15:20:16.218580 5083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:16.211020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:47071 (local address 127.3.211.254:40471)
0925 15:20:16.211057 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:16.211059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:16.211189 (+ 130us) server_negotiation.cc:407] Connection header received
0925 15:20:16.211245 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:16.211246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:16.211255 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:16.211316 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:16.211774 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.212524 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.213669 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.213876 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.213944 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:16.215172 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:16.215173 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:16.215174 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:16.215180 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:16.217093 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:16.217646 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:16.217647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:16.217648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:16.217729 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:16.218016 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:16.218017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:16.218018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:16.218169 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:16.218245 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:16.218408 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 15:20:16.218494 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab.
Entry for principal kudu/127.3.211.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab.
I20260925 15:20:16.239722 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.194:0
--local_ip_for_outbound_sockets=127.3.211.194
--webserver_interface=127.3.211.194
--webserver_port=0
--tserver_master_addrs=127.3.211.254:40471
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab
--principal=kudu/127.3.211.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:36825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:20:16.325603 5246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:16.325838 5246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:16.325866 5246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:16.327315 5246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:16.327378 5246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:16.327402 5246 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:20:16.327423 5246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.194
I20260925 15:20:16.329109 5246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:36825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab
--principal=kudu/127.3.211.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.211.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.211.254:40471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.194
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:16.329401 5246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:16.329671 5246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:16.335309 5251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:16.335314 5252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:16.335309 5254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:16.335387 5246 server_base.cc:1034] running on GCE node
I20260925 15:20:16.335803 5246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:16.336063 5246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:16.337230 5246 hybrid_clock.cc:648] HybridClock initialized: now 1790349616337231 us; error 45 us; skew 500 ppm
Sep 25 15:20:16 dist-test-slave-4bk3 krb5kdc[4989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349616, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:16.339960 5246 init.cc:377] Logged in from keytab as kudu/127.3.211.194@KRBTEST.COM (short username kudu)
I20260925 15:20:16.340538 5246 webserver.cc:460] Webserver started at http://127.3.211.194:42783/ using document root <none> and password file <none>
I20260925 15:20:16.340781 5246 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:16.340829 5246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:16.340966 5246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:16.342036 5246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/ts-1/data/instance:
uuid: "1217932a671e4457bf8c008451fafd01"
format_stamp: "Formatted at 2026-09-25 15:20:16 on dist-test-slave-4bk3"
server_key: "e2a69195d6ce9b04c1d469bec46e5976"
server_key_iv: "9d674727f41b8bba86dcdb656f61a184"
server_key_version: "encryptionkey@0"
I20260925 15:20:16.342402 5246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/ts-1/wal/instance:
uuid: "1217932a671e4457bf8c008451fafd01"
format_stamp: "Formatted at 2026-09-25 15:20:16 on dist-test-slave-4bk3"
server_key: "e2a69195d6ce9b04c1d469bec46e5976"
server_key_iv: "9d674727f41b8bba86dcdb656f61a184"
server_key_version: "encryptionkey@0"
I20260925 15:20:16.343786 5246 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:20:16.344630 5261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:16.344869 5246 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:20:16.344962 5246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/ts-1/wal
uuid: "1217932a671e4457bf8c008451fafd01"
format_stamp: "Formatted at 2026-09-25 15:20:16 on dist-test-slave-4bk3"
server_key: "e2a69195d6ce9b04c1d469bec46e5976"
server_key_iv: "9d674727f41b8bba86dcdb656f61a184"
server_key_version: "encryptionkey@0"
I20260925 15:20:16.345021 5246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:16.369118 5246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:16.371587 5246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:16.371716 5246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:16.372184 5246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:20:16.372220 5246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:16.372314 5246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:20:16.372388 5246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:16.380056 5246 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.194:46753
Sep 25 15:20:16 dist-test-slave-4bk3 krb5kdc[4989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349616, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.194@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:20:16.380146 5376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.194:46753 every 8 connection(s)
I20260925 15:20:16.380519 5246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
I20260925 15:20:16.383581 5083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:16.372821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:34957 (local address 127.3.211.254:40471)
0925 15:20:16.372894 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:16.372897 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:16.373125 (+ 228us) server_negotiation.cc:407] Connection header received
0925 15:20:16.373663 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:16.373665 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:16.373677 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:16.373766 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:16.374708 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.375578 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.377443 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.377795 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.378047 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:16.380769 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:16.380787 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:16.380787 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:16.380797 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:16.382167 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:16.382583 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:16.382584 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:16.382584 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:16.382653 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:16.382916 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:16.382917 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:16.382918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:16.383051 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:16.383129 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:16.383430 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 15:20:16.383513 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 15:20:16.383665 5270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:16.372794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:40471 (local address 127.3.211.194:34957)
0925 15:20:16.372987 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:16.373029 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:16.373567 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:16.373832 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:16.373835 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:16.374282 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:16.374591 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:16.374599 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.375714 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.375716 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:16.377183 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:16.377190 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.378064 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.378065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:16.378280 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:16.378911 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:16.378923 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:16.380654 (+ 1731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:16.382263 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:16.382269 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:16.382271 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:16.382505 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:16.382714 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:16.382715 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:16.382715 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:16.382817 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:16.383134 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:16.383138 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:16.383264 (+ 126us) client_negotiation.cc:769] Sending connection context
0925 15:20:16.383353 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 15:20:16.383486 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":82,"threads_started":1}
I20260925 15:20:16.384317 5377 heartbeater.cc:347] Connected to a master server at 127.3.211.254:40471
I20260925 15:20:16.384441 5377 heartbeater.cc:464] Registering TS with master...
I20260925 15:20:16.384657 5377 heartbeater.cc:511] Master 127.3.211.254:40471 requested a full tablet report, sending...
I20260925 15:20:16.385033 5033 ts_manager.cc:194] Registered new tserver with Master: 1217932a671e4457bf8c008451fafd01 (127.3.211.194:46753)
I20260925 15:20:16.385435 5033 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.194@KRBTEST.COM'} at 127.3.211.194:34957
I20260925 15:20:16.385681 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 5246
I20260925 15:20:16.385797 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/ts-1/wal/instance
I20260925 15:20:16.385991 3919 external_mini_cluster.cc:1428] Setting key c88cbbbffce4b12eebfe4394ee44735c
I20260925 15:20:16.393525 5083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:16.386178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:57649 (local address 127.3.211.254:40471)
0925 15:20:16.386235 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:16.386237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:16.386422 (+ 185us) server_negotiation.cc:407] Connection header received
0925 15:20:16.386463 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:16.386464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:16.386472 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:16.386526 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:16.386926 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.387408 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.388508 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.388860 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.388945 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:16.390170 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:16.390171 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:16.390171 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:16.390179 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:16.391727 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:16.392215 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:16.392216 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:16.392216 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:16.392279 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:16.392643 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:16.392644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:16.392645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:16.392831 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:16.392900 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:16.393368 (+ 468us) server_negotiation.cc:299] Negotiation successful
0925 15:20:16.393452 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 15:20:16.393621 5383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:16.385527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:40471 (local address 127.3.211.194:57649)
0925 15:20:16.386342 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:16.386354 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:16.386413 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:16.386532 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:16.386533 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:16.386748 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:16.386832 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:16.386835 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.387519 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.387520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:16.388419 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:16.388421 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.388952 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.388952 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:16.389015 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:16.389506 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:16.389513 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:16.390029 (+ 516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:16.391831 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:16.391832 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:16.391833 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:16.392106 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:16.392400 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:16.392402 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:16.392402 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:16.392515 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:16.392920 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:16.392921 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:16.393011 (+ 90us) client_negotiation.cc:769] Sending connection context
0925 15:20:16.393341 (+ 330us) client_negotiation.cc:240] Negotiation successful
0925 15:20:16.393494 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":787,"thread_start_us":65,"threads_started":1}
WARNING: no policy specified for kudu/127.3.211.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab.
Entry for principal kudu/127.3.211.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab.
I20260925 15:20:16.415771 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.195:0
--local_ip_for_outbound_sockets=127.3.211.195
--webserver_interface=127.3.211.195
--webserver_port=0
--tserver_master_addrs=127.3.211.254:40471
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab
--principal=kudu/127.3.211.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:36825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:20:16.501396 5386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:16.501636 5386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:16.501665 5386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:16.502902 5386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:16.502948 5386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:16.502971 5386 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:20:16.502990 5386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.195
I20260925 15:20:16.504575 5386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:36825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab
--principal=kudu/127.3.211.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.211.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.211.254:40471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.195
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:16.504796 5386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:16.504985 5386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:16.510272 5391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:16.510274 5394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:16.510294 5386 server_base.cc:1034] running on GCE node
W20260925 15:20:16.510317 5392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:16.510779 5386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:16.511058 5386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:16.512269 5386 hybrid_clock.cc:648] HybridClock initialized: now 1790349616512263 us; error 46 us; skew 500 ppm
Sep 25 15:20:16 dist-test-slave-4bk3 krb5kdc[4989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349616, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:16.514966 5386 init.cc:377] Logged in from keytab as kudu/127.3.211.195@KRBTEST.COM (short username kudu)
I20260925 15:20:16.515470 5386 webserver.cc:460] Webserver started at http://127.3.211.195:33465/ using document root <none> and password file <none>
I20260925 15:20:16.515722 5386 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:16.515796 5386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:16.515949 5386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:16.517158 5386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/ts-2/data/instance:
uuid: "12a76517a01541d0ab63360f1b9c04c5"
format_stamp: "Formatted at 2026-09-25 15:20:16 on dist-test-slave-4bk3"
server_key: "6ba021cede2650ca1259806174205ad1"
server_key_iv: "9c829de1e63dd962df0447cf4f60c316"
server_key_version: "encryptionkey@0"
I20260925 15:20:16.517553 5386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/ts-2/wal/instance:
uuid: "12a76517a01541d0ab63360f1b9c04c5"
format_stamp: "Formatted at 2026-09-25 15:20:16 on dist-test-slave-4bk3"
server_key: "6ba021cede2650ca1259806174205ad1"
server_key_iv: "9c829de1e63dd962df0447cf4f60c316"
server_key_version: "encryptionkey@0"
I20260925 15:20:16.519237 5386 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:20:16.520184 5401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:16.520434 5386 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:16.520573 5386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/ts-2/wal
uuid: "12a76517a01541d0ab63360f1b9c04c5"
format_stamp: "Formatted at 2026-09-25 15:20:16 on dist-test-slave-4bk3"
server_key: "6ba021cede2650ca1259806174205ad1"
server_key_iv: "9c829de1e63dd962df0447cf4f60c316"
server_key_version: "encryptionkey@0"
I20260925 15:20:16.520658 5386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:16.530880 5386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:16.533334 5386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:16.533460 5386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:16.533982 5386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:20:16.534063 5386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:16.534121 5386 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:20:16.534174 5386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 15:20:16 dist-test-slave-4bk3 krb5kdc[4989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349616, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.195@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:20:16.541915 5386 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.195:40347
I20260925 15:20:16.541996 5516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.195:40347 every 8 connection(s)
I20260925 15:20:16.542366 5386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
I20260925 15:20:16.544291 5083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:16.534714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:47847 (local address 127.3.211.254:40471)
0925 15:20:16.534758 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:16.534760 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:16.535001 (+ 241us) server_negotiation.cc:407] Connection header received
0925 15:20:16.535553 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:16.535554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:16.535567 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:16.535611 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:16.536591 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.537106 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.538261 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.538460 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.538556 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:16.541232 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:16.541234 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:16.541234 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:16.541242 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:16.542865 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:16.543304 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:16.543306 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:16.543306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:16.543368 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:16.543633 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:16.543634 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:16.543634 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:16.543761 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:16.543838 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:16.544118 (+ 280us) server_negotiation.cc:299] Negotiation successful
0925 15:20:16.544199 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 15:20:16.544523 5410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:16.534678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:40471 (local address 127.3.211.195:47847)
0925 15:20:16.534876 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:16.534914 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:16.535453 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:16.535717 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:16.535721 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:16.536140 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:16.536480 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:16.536486 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.537213 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.537214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:16.538156 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:16.538159 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.538568 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.538569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:16.538750 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:16.539426 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:16.539437 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:16.541136 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:16.542963 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:16.542969 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:16.542971 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:16.543208 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:16.543435 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:16.543436 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:16.543437 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:16.543555 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:16.543828 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:16.543834 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:16.543977 (+ 143us) client_negotiation.cc:769] Sending connection context
0925 15:20:16.544092 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 15:20:16.544257 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":107,"threads_started":1}
I20260925 15:20:16.545038 5517 heartbeater.cc:347] Connected to a master server at 127.3.211.254:40471
I20260925 15:20:16.545145 5517 heartbeater.cc:464] Registering TS with master...
I20260925 15:20:16.545331 5517 heartbeater.cc:511] Master 127.3.211.254:40471 requested a full tablet report, sending...
I20260925 15:20:16.545732 5033 ts_manager.cc:194] Registered new tserver with Master: 12a76517a01541d0ab63360f1b9c04c5 (127.3.211.195:40347)
I20260925 15:20:16.546252 5033 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.195@KRBTEST.COM'} at 127.3.211.195:47847
I20260925 15:20:16.551373 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 5386
I20260925 15:20:16.551473 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/ts-2/wal/instance
I20260925 15:20:16.551679 3919 external_mini_cluster.cc:1428] Setting key 418a0be4f40c7ae03873aa4b5e0a70fb
I20260925 15:20:16.552749 5083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:16.546159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:46587 (local address 127.3.211.254:40471)
0925 15:20:16.546200 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:16.546202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:16.546318 (+ 116us) server_negotiation.cc:407] Connection header received
0925 15:20:16.546417 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:16.546418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:16.546427 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:16.546483 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:16.546872 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.547438 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.548285 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.548570 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.548665 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:16.549691 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:16.549693 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:16.549693 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:16.549699 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:16.551023 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:16.551620 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:16.551620 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:16.551621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:16.551692 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:16.551962 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:16.551963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:16.551964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:16.552242 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:16.552365 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:16.552558 (+ 193us) server_negotiation.cc:299] Negotiation successful
0925 15:20:16.552656 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 15:20:16.552765 5523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:16.546087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:40471 (local address 127.3.211.195:46587)
0925 15:20:16.546241 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:16.546252 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:16.546334 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:16.546489 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:16.546490 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:16.546708 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:16.546785 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:16.546787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.547560 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.547561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:16.548165 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:16.548168 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.548692 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.548693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:16.548784 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:16.549267 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:16.549271 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:16.549563 (+ 292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:16.551207 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:16.551208 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:16.551208 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:16.551474 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:16.551799 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:16.551800 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:16.551800 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:16.551871 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:16.552374 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:16.552375 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:16.552428 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 15:20:16.552516 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 15:20:16.552648 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":102,"threads_started":1}
I20260925 15:20:16.553089 3919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 15:20:16 dist-test-slave-4bk3 krb5kdc[4989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349616, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 15:20:16 dist-test-slave-4bk3 krb5kdc[4989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349616, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:20:16.577874 5083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:16.569490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53546 (local address 127.3.211.254:40471)
0925 15:20:16.569594 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:16.569596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:16.569606 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:20:16.569638 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:16.569639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:16.569648 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:16.569754 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:16.570383 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.570910 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.571653 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.571845 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.571937 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:16.574515 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:16.574517 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:16.574517 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:16.574525 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:16.575926 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:16.576550 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:16.576552 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:16.576553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:16.576666 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:16.576981 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:16.576982 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:16.576983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:16.577228 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:16.577327 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:16.577657 (+ 330us) server_negotiation.cc:299] Negotiation successful
0925 15:20:16.577783 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 15:20:16.580310 5033 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53546:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:20:16.581956 5033 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:20:16.588951 5536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:16.584540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.194:46753 (local address 127.0.0.1:57084)
0925 15:20:16.584769 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:16.584792 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:16.584861 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:16.585235 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:16.585239 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:16.585241 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:20:16.585349 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:16.585352 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.586407 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.586408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:16.587664 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:16.587667 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.588568 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.588569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:16.588669 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:16.588684 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 15:20:16.588788 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 15:20:16.588852 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":3,"thread_start_us":95,"threads_started":1}
I20260925 15:20:16.588956 5539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:16.584890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57084 (local address 127.3.211.194:46753)
0925 15:20:16.585044 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:16.585049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:16.585059 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:20:16.585109 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:16.585111 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:16.585125 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:16.585231 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:20:16.585430 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.586312 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.587761 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.588474 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.588557 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:16.588623 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:16.588806 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 15:20:16.588849 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":80,"threads_started":1}
I20260925 15:20:16.589428 5535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:16.584645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:35851 (local address 127.0.0.1:55864)
0925 15:20:16.584956 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:16.584964 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:16.585002 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:16.585646 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:16.585647 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:16.585649 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:20:16.585689 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:16.585690 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.586924 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.586925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:16.587940 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:16.587943 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.589150 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.589151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:16.589226 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:16.589236 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 15:20:16.589303 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 15:20:16.589345 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":6,"thread_start_us":96,"threads_started":1}
I20260925 15:20:16.589501 5538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:16.584890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55864 (local address 127.3.211.193:35851)
0925 15:20:16.585344 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:16.585348 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:16.585357 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:20:16.585393 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:16.585395 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:16.585412 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:16.585556 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:20:16.585771 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.586833 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.588037 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.588886 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.588969 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:16.589111 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:16.589336 (+ 225us) server_negotiation.cc:299] Negotiation successful
0925 15:20:16.589388 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":83,"threads_started":1}
I20260925 15:20:16.589632 5534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:16.584540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.195:40347 (local address 127.0.0.1:42020)
0925 15:20:16.584750 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:16.584792 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:16.584861 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:16.585431 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:16.585432 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:16.585434 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:20:16.585472 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:16.585473 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.586803 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.586804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:16.587835 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:16.587836 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.589150 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.589151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:16.589226 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:16.589234 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 15:20:16.589290 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 15:20:16.589346 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":7,"thread_start_us":99,"threads_started":1}
I20260925 15:20:16.590037 5537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:16.584682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42020 (local address 127.3.211.195:40347)
0925 15:20:16.585082 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:16.585085 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:16.585095 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:20:16.585126 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:16.585127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:16.585139 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:16.585228 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:20:16.585554 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.586706 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.587928 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.588886 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.588969 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:16.589895 (+ 926us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:16.589919 (+ 24us) server_negotiation.cc:299] Negotiation successful
0925 15:20:16.589956 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":79,"threads_started":1}
I20260925 15:20:16.590148 5171 tablet_service.cc:1467] Processing CreateTablet for tablet 1dd3a56628664caa85aa24df5c375baa (DEFAULT_TABLE table=test-table [id=9148df6b2364464bbb2530d18fdcce22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:20:16.590092 5311 tablet_service.cc:1467] Processing CreateTablet for tablet 1dd3a56628664caa85aa24df5c375baa (DEFAULT_TABLE table=test-table [id=9148df6b2364464bbb2530d18fdcce22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:20:16.590492 5171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dd3a56628664caa85aa24df5c375baa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:16.590492 5311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dd3a56628664caa85aa24df5c375baa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:16.590580 5451 tablet_service.cc:1467] Processing CreateTablet for tablet 1dd3a56628664caa85aa24df5c375baa (DEFAULT_TABLE table=test-table [id=9148df6b2364464bbb2530d18fdcce22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:20:16.590818 5451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dd3a56628664caa85aa24df5c375baa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:16.593621 5540 tablet_bootstrap.cc:492] T 1dd3a56628664caa85aa24df5c375baa P 12a76517a01541d0ab63360f1b9c04c5: Bootstrap starting.
I20260925 15:20:16.594269 5541 tablet_bootstrap.cc:492] T 1dd3a56628664caa85aa24df5c375baa P 1217932a671e4457bf8c008451fafd01: Bootstrap starting.
I20260925 15:20:16.594514 5542 tablet_bootstrap.cc:492] T 1dd3a56628664caa85aa24df5c375baa P aecdd2c927464303a8a661264435dc61: Bootstrap starting.
I20260925 15:20:16.594871 5540 tablet_bootstrap.cc:654] T 1dd3a56628664caa85aa24df5c375baa P 12a76517a01541d0ab63360f1b9c04c5: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:16.595150 5542 tablet_bootstrap.cc:654] T 1dd3a56628664caa85aa24df5c375baa P aecdd2c927464303a8a661264435dc61: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:16.595247 5540 log.cc:824] T 1dd3a56628664caa85aa24df5c375baa P 12a76517a01541d0ab63360f1b9c04c5: Log is configured to *not* fsync() on all Append() calls
I20260925 15:20:16.595499 5542 log.cc:824] T 1dd3a56628664caa85aa24df5c375baa P aecdd2c927464303a8a661264435dc61: Log is configured to *not* fsync() on all Append() calls
I20260925 15:20:16.596218 5541 tablet_bootstrap.cc:654] T 1dd3a56628664caa85aa24df5c375baa P 1217932a671e4457bf8c008451fafd01: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:16.596362 5540 tablet_bootstrap.cc:492] T 1dd3a56628664caa85aa24df5c375baa P 12a76517a01541d0ab63360f1b9c04c5: No bootstrap required, opened a new log
I20260925 15:20:16.596438 5540 ts_tablet_manager.cc:1397] T 1dd3a56628664caa85aa24df5c375baa P 12a76517a01541d0ab63360f1b9c04c5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 15:20:16.596596 5541 log.cc:824] T 1dd3a56628664caa85aa24df5c375baa P 1217932a671e4457bf8c008451fafd01: Log is configured to *not* fsync() on all Append() calls
I20260925 15:20:16.597390 5541 tablet_bootstrap.cc:492] T 1dd3a56628664caa85aa24df5c375baa P 1217932a671e4457bf8c008451fafd01: No bootstrap required, opened a new log
I20260925 15:20:16.597432 5542 tablet_bootstrap.cc:492] T 1dd3a56628664caa85aa24df5c375baa P aecdd2c927464303a8a661264435dc61: No bootstrap required, opened a new log
I20260925 15:20:16.597492 5542 ts_tablet_manager.cc:1397] T 1dd3a56628664caa85aa24df5c375baa P aecdd2c927464303a8a661264435dc61: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 15:20:16.597489 5541 ts_tablet_manager.cc:1397] T 1dd3a56628664caa85aa24df5c375baa P 1217932a671e4457bf8c008451fafd01: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 15:20:16.598714 5540 raft_consensus.cc:348] T 1dd3a56628664caa85aa24df5c375baa P 12a76517a01541d0ab63360f1b9c04c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a76517a01541d0ab63360f1b9c04c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 40347 } } peers { permanent_uuid: "1217932a671e4457bf8c008451fafd01" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 46753 } } peers { permanent_uuid: "aecdd2c927464303a8a661264435dc61" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 35851 } }
I20260925 15:20:16.599043 5540 raft_consensus.cc:374] T 1dd3a56628664caa85aa24df5c375baa P 12a76517a01541d0ab63360f1b9c04c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:16.599045 5541 raft_consensus.cc:348] T 1dd3a56628664caa85aa24df5c375baa P 1217932a671e4457bf8c008451fafd01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a76517a01541d0ab63360f1b9c04c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 40347 } } peers { permanent_uuid: "1217932a671e4457bf8c008451fafd01" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 46753 } } peers { permanent_uuid: "aecdd2c927464303a8a661264435dc61" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 35851 } }
I20260925 15:20:16.599265 5541 raft_consensus.cc:374] T 1dd3a56628664caa85aa24df5c375baa P 1217932a671e4457bf8c008451fafd01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:16.599236 5540 raft_consensus.cc:729] T 1dd3a56628664caa85aa24df5c375baa P 12a76517a01541d0ab63360f1b9c04c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12a76517a01541d0ab63360f1b9c04c5, State: Initialized, Role: FOLLOWER
I20260925 15:20:16.599354 5541 raft_consensus.cc:729] T 1dd3a56628664caa85aa24df5c375baa P 1217932a671e4457bf8c008451fafd01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1217932a671e4457bf8c008451fafd01, State: Initialized, Role: FOLLOWER
I20260925 15:20:16.599468 5540 consensus_queue.cc:260] T 1dd3a56628664caa85aa24df5c375baa P 12a76517a01541d0ab63360f1b9c04c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a76517a01541d0ab63360f1b9c04c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 40347 } } peers { permanent_uuid: "1217932a671e4457bf8c008451fafd01" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 46753 } } peers { permanent_uuid: "aecdd2c927464303a8a661264435dc61" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 35851 } }
I20260925 15:20:16.599491 5541 consensus_queue.cc:260] T 1dd3a56628664caa85aa24df5c375baa P 1217932a671e4457bf8c008451fafd01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a76517a01541d0ab63360f1b9c04c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 40347 } } peers { permanent_uuid: "1217932a671e4457bf8c008451fafd01" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 46753 } } peers { permanent_uuid: "aecdd2c927464303a8a661264435dc61" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 35851 } }
I20260925 15:20:16.599653 5542 raft_consensus.cc:348] T 1dd3a56628664caa85aa24df5c375baa P aecdd2c927464303a8a661264435dc61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a76517a01541d0ab63360f1b9c04c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 40347 } } peers { permanent_uuid: "1217932a671e4457bf8c008451fafd01" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 46753 } } peers { permanent_uuid: "aecdd2c927464303a8a661264435dc61" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 35851 } }
I20260925 15:20:16.599824 5540 ts_tablet_manager.cc:1428] T 1dd3a56628664caa85aa24df5c375baa P 12a76517a01541d0ab63360f1b9c04c5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:20:16.599829 5541 ts_tablet_manager.cc:1428] T 1dd3a56628664caa85aa24df5c375baa P 1217932a671e4457bf8c008451fafd01: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 15:20:16.599865 5517 heartbeater.cc:503] Master 127.3.211.254:40471 was elected leader, sending a full tablet report...
I20260925 15:20:16.599929 5377 heartbeater.cc:503] Master 127.3.211.254:40471 was elected leader, sending a full tablet report...
I20260925 15:20:16.600193 5542 raft_consensus.cc:374] T 1dd3a56628664caa85aa24df5c375baa P aecdd2c927464303a8a661264435dc61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:16.600263 5542 raft_consensus.cc:729] T 1dd3a56628664caa85aa24df5c375baa P aecdd2c927464303a8a661264435dc61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aecdd2c927464303a8a661264435dc61, State: Initialized, Role: FOLLOWER
I20260925 15:20:16.600435 5542 consensus_queue.cc:260] T 1dd3a56628664caa85aa24df5c375baa P aecdd2c927464303a8a661264435dc61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a76517a01541d0ab63360f1b9c04c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 40347 } } peers { permanent_uuid: "1217932a671e4457bf8c008451fafd01" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 46753 } } peers { permanent_uuid: "aecdd2c927464303a8a661264435dc61" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 35851 } }
I20260925 15:20:16.600765 5542 ts_tablet_manager.cc:1428] T 1dd3a56628664caa85aa24df5c375baa P aecdd2c927464303a8a661264435dc61: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:20:16.600937 5237 heartbeater.cc:503] Master 127.3.211.254:40471 was elected leader, sending a full tablet report...
I20260925 15:20:16.628999 5548 raft_consensus.cc:482] T 1dd3a56628664caa85aa24df5c375baa P aecdd2c927464303a8a661264435dc61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:20:16.629191 5548 raft_consensus.cc:504] T 1dd3a56628664caa85aa24df5c375baa P aecdd2c927464303a8a661264435dc61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a76517a01541d0ab63360f1b9c04c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 40347 } } peers { permanent_uuid: "1217932a671e4457bf8c008451fafd01" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 46753 } } peers { permanent_uuid: "aecdd2c927464303a8a661264435dc61" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 35851 } }
I20260925 15:20:16.629789 5548 leader_election.cc:288] T 1dd3a56628664caa85aa24df5c375baa P aecdd2c927464303a8a661264435dc61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12a76517a01541d0ab63360f1b9c04c5 (127.3.211.195:40347), 1217932a671e4457bf8c008451fafd01 (127.3.211.194:46753)
I20260925 15:20:16.630978 5546 raft_consensus.cc:482] T 1dd3a56628664caa85aa24df5c375baa P 12a76517a01541d0ab63360f1b9c04c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260925 15:20:16.631107 5378 tablet.cc:2369] T 1dd3a56628664caa85aa24df5c375baa P 1217932a671e4457bf8c008451fafd01: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:20:16.631117 5546 raft_consensus.cc:504] T 1dd3a56628664caa85aa24df5c375baa P 12a76517a01541d0ab63360f1b9c04c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a76517a01541d0ab63360f1b9c04c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 40347 } } peers { permanent_uuid: "1217932a671e4457bf8c008451fafd01" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 46753 } } peers { permanent_uuid: "aecdd2c927464303a8a661264435dc61" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 35851 } }
I20260925 15:20:16.631518 5546 leader_election.cc:288] T 1dd3a56628664caa85aa24df5c375baa P 12a76517a01541d0ab63360f1b9c04c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1217932a671e4457bf8c008451fafd01 (127.3.211.194:46753), aecdd2c927464303a8a661264435dc61 (127.3.211.193:35851)
I20260925 15:20:16.633796 5130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:16.629979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.194:46753 (local address 127.3.211.193:57963)
0925 15:20:16.630053 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:16.630067 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:16.630125 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:16.630424 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:16.630426 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:16.630428 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:20:16.630539 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:16.630541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.631484 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.631485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:16.632763 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:16.632765 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.633495 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.633496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:16.633600 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:16.633604 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:20:16.633655 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 15:20:16.633712 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":3}
I20260925 15:20:16.634222 5537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:16.630032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:36943 (local address 127.3.211.195:40347)
0925 15:20:16.630090 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:16.630092 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:16.630280 (+ 188us) server_negotiation.cc:407] Connection header received
0925 15:20:16.630309 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:16.630311 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:16.630319 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:16.630379 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:20:16.630640 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.631388 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.632836 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.633381 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.633423 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:16.633434 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:16.634096 (+ 662us) server_negotiation.cc:299] Negotiation successful
0925 15:20:16.634140 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 15:20:16.634264 5539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:16.630158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:57963 (local address 127.3.211.194:46753)
0925 15:20:16.630230 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:16.630232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:16.630241 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:20:16.630275 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:16.630276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:16.630287 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:16.630359 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:20:16.630639 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.631389 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.632855 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.633395 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.633458 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:16.633466 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:16.634157 (+ 691us) server_negotiation.cc:299] Negotiation successful
0925 15:20:16.634203 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 15:20:16.634567 5331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dd3a56628664caa85aa24df5c375baa" candidate_uuid: "aecdd2c927464303a8a661264435dc61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1217932a671e4457bf8c008451fafd01" is_pre_election: true
I20260925 15:20:16.634702 5549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:16.629987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.195:40347 (local address 127.3.211.193:36943)
0925 15:20:16.630187 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:16.630196 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:16.630268 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:16.630424 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:16.630426 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:16.630428 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:20:16.630539 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:16.630541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.631510 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.631511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:16.632746 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:16.632749 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.633492 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.633493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:16.633983 (+ 490us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:16.633988 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:20:16.634536 (+ 548us) client_negotiation.cc:240] Negotiation successful
0925 15:20:16.634583 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":42,"thread_start_us":129,"threads_started":1}
I20260925 15:20:16.634738 5331 raft_consensus.cc:2454] T 1dd3a56628664caa85aa24df5c375baa P 1217932a671e4457bf8c008451fafd01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aecdd2c927464303a8a661264435dc61 in term 0.
I20260925 15:20:16.635058 5125 leader_election.cc:302] T 1dd3a56628664caa85aa24df5c375baa P aecdd2c927464303a8a661264435dc61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1217932a671e4457bf8c008451fafd01, aecdd2c927464303a8a661264435dc61; no voters:
I20260925 15:20:16.635141 5471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dd3a56628664caa85aa24df5c375baa" candidate_uuid: "aecdd2c927464303a8a661264435dc61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12a76517a01541d0ab63360f1b9c04c5" is_pre_election: true
I20260925 15:20:16.635187 5548 raft_consensus.cc:2781] T 1dd3a56628664caa85aa24df5c375baa P aecdd2c927464303a8a661264435dc61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:20:16.635232 5548 raft_consensus.cc:482] T 1dd3a56628664caa85aa24df5c375baa P aecdd2c927464303a8a661264435dc61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:20:16.635255 5548 raft_consensus.cc:3037] T 1dd3a56628664caa85aa24df5c375baa P aecdd2c927464303a8a661264435dc61 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:16.635242 5471 raft_consensus.cc:2454] T 1dd3a56628664caa85aa24df5c375baa P 12a76517a01541d0ab63360f1b9c04c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aecdd2c927464303a8a661264435dc61 in term 0.
I20260925 15:20:16.635913 5538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:16.631806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:41281 (local address 127.3.211.193:35851)
0925 15:20:16.631849 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:16.631850 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:16.632016 (+ 166us) server_negotiation.cc:407] Connection header received
0925 15:20:16.632037 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:16.632037 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:16.632046 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:16.632087 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:20:16.632468 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.633105 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.634573 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.635266 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.635333 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:16.635342 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:16.635802 (+ 460us) server_negotiation.cc:299] Negotiation successful
0925 15:20:16.635852 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 15:20:16.635926 5410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:16.631718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:35851 (local address 127.3.211.195:41281)
0925 15:20:16.631901 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:16.631912 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:16.631959 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:16.632310 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:16.632311 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:16.632312 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:20:16.632390 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:16.632392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.633203 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.633204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:16.634341 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:16.634344 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.635586 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.635587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:16.635695 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:16.635699 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:20:16.635787 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 15:20:16.635846 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167}
I20260925 15:20:16.636113 5550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:16.631822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.194:46753 (local address 127.3.211.195:55279)
0925 15:20:16.632007 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:16.632016 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:16.632062 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:16.632505 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:16.632506 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:16.632508 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:20:16.632579 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:16.632581 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.633468 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.633469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:16.634601 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:16.634604 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.635366 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.635368 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:16.635933 (+ 565us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:16.635937 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:20:16.635972 (+ 35us) client_negotiation.cc:240] Negotiation successful
0925 15:20:16.636013 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":111,"threads_started":1}
I20260925 15:20:16.636404 5551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:16.632169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:55279 (local address 127.3.211.194:46753)
0925 15:20:16.632374 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:16.632376 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:16.632384 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:20:16.632404 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:16.632405 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:16.632414 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:16.632636 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:20:16.632664 (+ 28us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.633368 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.634702 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.635269 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.636203 (+ 934us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:16.636216 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:16.636235 (+ 19us) server_negotiation.cc:299] Negotiation successful
0925 15:20:16.636283 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":107,"threads_started":1}
I20260925 15:20:16.636397 5191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dd3a56628664caa85aa24df5c375baa" candidate_uuid: "12a76517a01541d0ab63360f1b9c04c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aecdd2c927464303a8a661264435dc61" is_pre_election: true
I20260925 15:20:16.636423 5548 raft_consensus.cc:504] T 1dd3a56628664caa85aa24df5c375baa P aecdd2c927464303a8a661264435dc61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a76517a01541d0ab63360f1b9c04c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 40347 } } peers { permanent_uuid: "1217932a671e4457bf8c008451fafd01" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 46753 } } peers { permanent_uuid: "aecdd2c927464303a8a661264435dc61" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 35851 } }
I20260925 15:20:16.636505 5191 raft_consensus.cc:2379] T 1dd3a56628664caa85aa24df5c375baa P aecdd2c927464303a8a661264435dc61 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 12a76517a01541d0ab63360f1b9c04c5 in current term 1: Already voted for candidate aecdd2c927464303a8a661264435dc61 in this term.
I20260925 15:20:16.636555 5548 leader_election.cc:288] T 1dd3a56628664caa85aa24df5c375baa P aecdd2c927464303a8a661264435dc61 [CANDIDATE]: Term 1 election: Requested vote from peers 12a76517a01541d0ab63360f1b9c04c5 (127.3.211.195:40347), 1217932a671e4457bf8c008451fafd01 (127.3.211.194:46753)
I20260925 15:20:16.636745 5471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dd3a56628664caa85aa24df5c375baa" candidate_uuid: "aecdd2c927464303a8a661264435dc61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12a76517a01541d0ab63360f1b9c04c5"
I20260925 15:20:16.636801 5331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dd3a56628664caa85aa24df5c375baa" candidate_uuid: "aecdd2c927464303a8a661264435dc61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1217932a671e4457bf8c008451fafd01"
I20260925 15:20:16.636837 5471 raft_consensus.cc:3037] T 1dd3a56628664caa85aa24df5c375baa P 12a76517a01541d0ab63360f1b9c04c5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:16.636888 5331 raft_consensus.cc:3037] T 1dd3a56628664caa85aa24df5c375baa P 1217932a671e4457bf8c008451fafd01 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:16.637025 5330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dd3a56628664caa85aa24df5c375baa" candidate_uuid: "12a76517a01541d0ab63360f1b9c04c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1217932a671e4457bf8c008451fafd01" is_pre_election: true
I20260925 15:20:16.637342 5405 leader_election.cc:302] T 1dd3a56628664caa85aa24df5c375baa P 12a76517a01541d0ab63360f1b9c04c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 12a76517a01541d0ab63360f1b9c04c5; no voters: 1217932a671e4457bf8c008451fafd01, aecdd2c927464303a8a661264435dc61
I20260925 15:20:16.637795 5471 raft_consensus.cc:2454] T 1dd3a56628664caa85aa24df5c375baa P 12a76517a01541d0ab63360f1b9c04c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aecdd2c927464303a8a661264435dc61 in term 1.
I20260925 15:20:16.637935 5546 raft_consensus.cc:2726] T 1dd3a56628664caa85aa24df5c375baa P 12a76517a01541d0ab63360f1b9c04c5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 15:20:16.637754 5331 raft_consensus.cc:2454] T 1dd3a56628664caa85aa24df5c375baa P 1217932a671e4457bf8c008451fafd01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aecdd2c927464303a8a661264435dc61 in term 1.
I20260925 15:20:16.638291 5124 leader_election.cc:302] T 1dd3a56628664caa85aa24df5c375baa P aecdd2c927464303a8a661264435dc61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12a76517a01541d0ab63360f1b9c04c5, aecdd2c927464303a8a661264435dc61; no voters:
I20260925 15:20:16.638403 5548 raft_consensus.cc:2781] T 1dd3a56628664caa85aa24df5c375baa P aecdd2c927464303a8a661264435dc61 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:20:16.638588 5548 raft_consensus.cc:686] T 1dd3a56628664caa85aa24df5c375baa P aecdd2c927464303a8a661264435dc61 [term 1 LEADER]: Becoming Leader. State: Replica: aecdd2c927464303a8a661264435dc61, State: Running, Role: LEADER
I20260925 15:20:16.638674 5548 consensus_queue.cc:237] T 1dd3a56628664caa85aa24df5c375baa P aecdd2c927464303a8a661264435dc61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a76517a01541d0ab63360f1b9c04c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 40347 } } peers { permanent_uuid: "1217932a671e4457bf8c008451fafd01" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 46753 } } peers { permanent_uuid: "aecdd2c927464303a8a661264435dc61" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 35851 } }
I20260925 15:20:16.639962 5034 catalog_manager.cc:5495] T 1dd3a56628664caa85aa24df5c375baa P aecdd2c927464303a8a661264435dc61 reported cstate change: term changed from 0 to 1, leader changed from <none> to aecdd2c927464303a8a661264435dc61 (127.3.211.193). New cstate: current_term: 1 leader_uuid: "aecdd2c927464303a8a661264435dc61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12a76517a01541d0ab63360f1b9c04c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 40347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1217932a671e4457bf8c008451fafd01" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 46753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aecdd2c927464303a8a661264435dc61" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 35851 } health_report { overall_health: HEALTHY } } }
I20260925 15:20:16.655683 5538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:16.652460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55868 (local address 127.3.211.193:35851)
0925 15:20:16.652531 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:16.652532 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:16.652602 (+ 70us) server_negotiation.cc:407] Connection header received
0925 15:20:16.652670 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:16.652671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:16.652688 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:16.652751 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:20:16.652901 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.653379 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.654430 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.654628 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.654699 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:16.655026 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:20:16.655201 (+ 175us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:20:16.655334 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:16.655455 (+ 121us) server_negotiation.cc:299] Negotiation successful
0925 15:20:16.655489 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 15:20:16.658288 5331 raft_consensus.cc:1261] T 1dd3a56628664caa85aa24df5c375baa P 1217932a671e4457bf8c008451fafd01 [term 1 FOLLOWER]: Refusing update from remote peer aecdd2c927464303a8a661264435dc61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:20:16.658550 5471 raft_consensus.cc:1261] T 1dd3a56628664caa85aa24df5c375baa P 12a76517a01541d0ab63360f1b9c04c5 [term 1 FOLLOWER]: Refusing update from remote peer aecdd2c927464303a8a661264435dc61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:20:16.658653 5552 consensus_queue.cc:1035] T 1dd3a56628664caa85aa24df5c375baa P aecdd2c927464303a8a661264435dc61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1217932a671e4457bf8c008451fafd01" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 46753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:20:16.658939 5552 consensus_queue.cc:1035] T 1dd3a56628664caa85aa24df5c375baa P aecdd2c927464303a8a661264435dc61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12a76517a01541d0ab63360f1b9c04c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 40347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:20:16.661924 5555 mvcc.cc:205] Tried to move back new op lower bound from 7333272029828927488 to 7333272029752705024. Current Snapshot: MvccSnapshot[applied={T|T < 7333272029828927488}]
I20260925 15:20:16.662212 5556 mvcc.cc:205] Tried to move back new op lower bound from 7333272029828927488 to 7333272029752705024. Current Snapshot: MvccSnapshot[applied={T|T < 7333272029828927488}]
I20260925 15:20:16.664484 5559 mvcc.cc:205] Tried to move back new op lower bound from 7333272029828927488 to 7333272029752705024. Current Snapshot: MvccSnapshot[applied={T|T < 7333272029828927488 or (T in {7333272029828927488})}]
I20260925 15:20:16.666858 5034 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:20:16.667064 5034 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:20:16.668376 5034 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 980593882e994988bb22962652e15a9d: Sending DeleteTablet for 3 replicas of tablet 1dd3a56628664caa85aa24df5c375baa
I20260925 15:20:16.669245 5451 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dd3a56628664caa85aa24df5c375baa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:20:16 UTC) from {username='kudu', principal='kudu/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:42020
I20260925 15:20:16.669245 5171 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dd3a56628664caa85aa24df5c375baa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:20:16 UTC) from {username='kudu', principal='kudu/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:55864
I20260925 15:20:16.669874 5034 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53546:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:20:16.670013 5034 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:20:16.670410 5564 tablet_replica.cc:330] T 1dd3a56628664caa85aa24df5c375baa P 12a76517a01541d0ab63360f1b9c04c5: stopping tablet replica
I20260925 15:20:16.670547 5564 raft_consensus.cc:2229] T 1dd3a56628664caa85aa24df5c375baa P 12a76517a01541d0ab63360f1b9c04c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:20:16.670702 5311 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dd3a56628664caa85aa24df5c375baa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:20:16 UTC) from {username='kudu', principal='kudu/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:57084
I20260925 15:20:16.670753 5564 raft_consensus.cc:2258] T 1dd3a56628664caa85aa24df5c375baa P 12a76517a01541d0ab63360f1b9c04c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:20:16.670967 5566 tablet_replica.cc:330] T 1dd3a56628664caa85aa24df5c375baa P 1217932a671e4457bf8c008451fafd01: stopping tablet replica
I20260925 15:20:16.671129 5566 raft_consensus.cc:2229] T 1dd3a56628664caa85aa24df5c375baa P 1217932a671e4457bf8c008451fafd01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:20:16.671291 5566 raft_consensus.cc:2258] T 1dd3a56628664caa85aa24df5c375baa P 1217932a671e4457bf8c008451fafd01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:20:16.671288 5564 ts_tablet_manager.cc:1905] T 1dd3a56628664caa85aa24df5c375baa P 12a76517a01541d0ab63360f1b9c04c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:20:16.672189 5171 tablet_service.cc:1467] Processing CreateTablet for tablet fdba6b2348df4b09b5bef67363ce3af2 (DEFAULT_TABLE table=test-table [id=dfa89c7cad1a4af4bf2ef208d73d1e99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:20:16.672225 5451 tablet_service.cc:1467] Processing CreateTablet for tablet fdba6b2348df4b09b5bef67363ce3af2 (DEFAULT_TABLE table=test-table [id=dfa89c7cad1a4af4bf2ef208d73d1e99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:20:16.672364 5171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdba6b2348df4b09b5bef67363ce3af2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:16.672376 5451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdba6b2348df4b09b5bef67363ce3af2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:16.673619 5540 tablet_bootstrap.cc:492] T fdba6b2348df4b09b5bef67363ce3af2 P 12a76517a01541d0ab63360f1b9c04c5: Bootstrap starting.
I20260925 15:20:16.673792 5311 tablet_service.cc:1467] Processing CreateTablet for tablet fdba6b2348df4b09b5bef67363ce3af2 (DEFAULT_TABLE table=test-table [id=dfa89c7cad1a4af4bf2ef208d73d1e99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:20:16.673834 5564 ts_tablet_manager.cc:1918] T 1dd3a56628664caa85aa24df5c375baa P 12a76517a01541d0ab63360f1b9c04c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:20:16.673899 5564 log.cc:1196] T 1dd3a56628664caa85aa24df5c375baa P 12a76517a01541d0ab63360f1b9c04c5: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/ts-2/wal/wals/1dd3a56628664caa85aa24df5c375baa
I20260925 15:20:16.673913 5311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdba6b2348df4b09b5bef67363ce3af2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:16.673920 5542 tablet_bootstrap.cc:492] T fdba6b2348df4b09b5bef67363ce3af2 P aecdd2c927464303a8a661264435dc61: Bootstrap starting.
I20260925 15:20:16.674093 5540 tablet_bootstrap.cc:654] T fdba6b2348df4b09b5bef67363ce3af2 P 12a76517a01541d0ab63360f1b9c04c5: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:16.674180 5564 ts_tablet_manager.cc:1939] T 1dd3a56628664caa85aa24df5c375baa P 12a76517a01541d0ab63360f1b9c04c5: Deleting consensus metadata
I20260925 15:20:16.674301 5542 tablet_bootstrap.cc:654] T fdba6b2348df4b09b5bef67363ce3af2 P aecdd2c927464303a8a661264435dc61: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:16.674592 5020 catalog_manager.cc:4841] TS 12a76517a01541d0ab63360f1b9c04c5 (127.3.211.195:40347): tablet 1dd3a56628664caa85aa24df5c375baa (table test-table [id=9148df6b2364464bbb2530d18fdcce22]) successfully deleted
I20260925 15:20:16.674784 5565 tablet_replica.cc:330] T 1dd3a56628664caa85aa24df5c375baa P aecdd2c927464303a8a661264435dc61: stopping tablet replica
I20260925 15:20:16.674894 5565 raft_consensus.cc:2229] T 1dd3a56628664caa85aa24df5c375baa P aecdd2c927464303a8a661264435dc61 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:20:16.675027 5565 raft_consensus.cc:2258] T 1dd3a56628664caa85aa24df5c375baa P aecdd2c927464303a8a661264435dc61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:20:16.675379 5541 tablet_bootstrap.cc:492] T fdba6b2348df4b09b5bef67363ce3af2 P 1217932a671e4457bf8c008451fafd01: Bootstrap starting.
I20260925 15:20:16.675801 5541 tablet_bootstrap.cc:654] T fdba6b2348df4b09b5bef67363ce3af2 P 1217932a671e4457bf8c008451fafd01: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:16.675908 5566 ts_tablet_manager.cc:1905] T 1dd3a56628664caa85aa24df5c375baa P 1217932a671e4457bf8c008451fafd01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:20:16.676450 5541 tablet_bootstrap.cc:492] T fdba6b2348df4b09b5bef67363ce3af2 P 1217932a671e4457bf8c008451fafd01: No bootstrap required, opened a new log
I20260925 15:20:16.676497 5541 ts_tablet_manager.cc:1397] T fdba6b2348df4b09b5bef67363ce3af2 P 1217932a671e4457bf8c008451fafd01: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:16.676651 5541 raft_consensus.cc:348] T fdba6b2348df4b09b5bef67363ce3af2 P 1217932a671e4457bf8c008451fafd01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aecdd2c927464303a8a661264435dc61" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 35851 } } peers { permanent_uuid: "1217932a671e4457bf8c008451fafd01" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 46753 } } peers { permanent_uuid: "12a76517a01541d0ab63360f1b9c04c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 40347 } }
I20260925 15:20:16.676719 5541 raft_consensus.cc:374] T fdba6b2348df4b09b5bef67363ce3af2 P 1217932a671e4457bf8c008451fafd01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:16.676739 5541 raft_consensus.cc:729] T fdba6b2348df4b09b5bef67363ce3af2 P 1217932a671e4457bf8c008451fafd01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1217932a671e4457bf8c008451fafd01, State: Initialized, Role: FOLLOWER
I20260925 15:20:16.676764 5542 tablet_bootstrap.cc:492] T fdba6b2348df4b09b5bef67363ce3af2 P aecdd2c927464303a8a661264435dc61: No bootstrap required, opened a new log
I20260925 15:20:16.676807 5542 ts_tablet_manager.cc:1397] T fdba6b2348df4b09b5bef67363ce3af2 P aecdd2c927464303a8a661264435dc61: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 15:20:16.676872 5541 consensus_queue.cc:260] T fdba6b2348df4b09b5bef67363ce3af2 P 1217932a671e4457bf8c008451fafd01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aecdd2c927464303a8a661264435dc61" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 35851 } } peers { permanent_uuid: "1217932a671e4457bf8c008451fafd01" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 46753 } } peers { permanent_uuid: "12a76517a01541d0ab63360f1b9c04c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 40347 } }
I20260925 15:20:16.677026 5541 ts_tablet_manager.cc:1428] T fdba6b2348df4b09b5bef67363ce3af2 P 1217932a671e4457bf8c008451fafd01: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:16.676978 5542 raft_consensus.cc:348] T fdba6b2348df4b09b5bef67363ce3af2 P aecdd2c927464303a8a661264435dc61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aecdd2c927464303a8a661264435dc61" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 35851 } } peers { permanent_uuid: "1217932a671e4457bf8c008451fafd01" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 46753 } } peers { permanent_uuid: "12a76517a01541d0ab63360f1b9c04c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 40347 } }
I20260925 15:20:16.677065 5542 raft_consensus.cc:374] T fdba6b2348df4b09b5bef67363ce3af2 P aecdd2c927464303a8a661264435dc61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:16.677089 5542 raft_consensus.cc:729] T fdba6b2348df4b09b5bef67363ce3af2 P aecdd2c927464303a8a661264435dc61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aecdd2c927464303a8a661264435dc61, State: Initialized, Role: FOLLOWER
I20260925 15:20:16.677134 5542 consensus_queue.cc:260] T fdba6b2348df4b09b5bef67363ce3af2 P aecdd2c927464303a8a661264435dc61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aecdd2c927464303a8a661264435dc61" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 35851 } } peers { permanent_uuid: "1217932a671e4457bf8c008451fafd01" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 46753 } } peers { permanent_uuid: "12a76517a01541d0ab63360f1b9c04c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 40347 } }
I20260925 15:20:16.677218 5542 ts_tablet_manager.cc:1428] T fdba6b2348df4b09b5bef67363ce3af2 P aecdd2c927464303a8a661264435dc61: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 15:20:16.677888 5565 ts_tablet_manager.cc:1905] T 1dd3a56628664caa85aa24df5c375baa P aecdd2c927464303a8a661264435dc61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:20:16.679461 5565 ts_tablet_manager.cc:1918] T 1dd3a56628664caa85aa24df5c375baa P aecdd2c927464303a8a661264435dc61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:20:16.679543 5565 log.cc:1196] T 1dd3a56628664caa85aa24df5c375baa P aecdd2c927464303a8a661264435dc61: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/ts-0/wal/wals/1dd3a56628664caa85aa24df5c375baa
I20260925 15:20:16.679808 5565 ts_tablet_manager.cc:1939] T 1dd3a56628664caa85aa24df5c375baa P aecdd2c927464303a8a661264435dc61: Deleting consensus metadata
I20260925 15:20:16.680112 5566 ts_tablet_manager.cc:1918] T 1dd3a56628664caa85aa24df5c375baa P 1217932a671e4457bf8c008451fafd01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:20:16.680217 5566 log.cc:1196] T 1dd3a56628664caa85aa24df5c375baa P 1217932a671e4457bf8c008451fafd01: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/ts-1/wal/wals/1dd3a56628664caa85aa24df5c375baa
I20260925 15:20:16.680291 5019 catalog_manager.cc:4841] TS aecdd2c927464303a8a661264435dc61 (127.3.211.193:35851): tablet 1dd3a56628664caa85aa24df5c375baa (table test-table [id=9148df6b2364464bbb2530d18fdcce22]) successfully deleted
I20260925 15:20:16.680526 5566 ts_tablet_manager.cc:1939] T 1dd3a56628664caa85aa24df5c375baa P 1217932a671e4457bf8c008451fafd01: Deleting consensus metadata
I20260925 15:20:16.680778 5540 tablet_bootstrap.cc:492] T fdba6b2348df4b09b5bef67363ce3af2 P 12a76517a01541d0ab63360f1b9c04c5: No bootstrap required, opened a new log
I20260925 15:20:16.680828 5540 ts_tablet_manager.cc:1397] T fdba6b2348df4b09b5bef67363ce3af2 P 12a76517a01541d0ab63360f1b9c04c5: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260925 15:20:16.681017 5021 catalog_manager.cc:4841] TS 1217932a671e4457bf8c008451fafd01 (127.3.211.194:46753): tablet 1dd3a56628664caa85aa24df5c375baa (table test-table [id=9148df6b2364464bbb2530d18fdcce22]) successfully deleted
I20260925 15:20:16.681010 5540 raft_consensus.cc:348] T fdba6b2348df4b09b5bef67363ce3af2 P 12a76517a01541d0ab63360f1b9c04c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aecdd2c927464303a8a661264435dc61" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 35851 } } peers { permanent_uuid: "1217932a671e4457bf8c008451fafd01" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 46753 } } peers { permanent_uuid: "12a76517a01541d0ab63360f1b9c04c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 40347 } }
I20260925 15:20:16.681092 5540 raft_consensus.cc:374] T fdba6b2348df4b09b5bef67363ce3af2 P 12a76517a01541d0ab63360f1b9c04c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:16.681108 5540 raft_consensus.cc:729] T fdba6b2348df4b09b5bef67363ce3af2 P 12a76517a01541d0ab63360f1b9c04c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12a76517a01541d0ab63360f1b9c04c5, State: Initialized, Role: FOLLOWER
I20260925 15:20:16.681192 5540 consensus_queue.cc:260] T fdba6b2348df4b09b5bef67363ce3af2 P 12a76517a01541d0ab63360f1b9c04c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aecdd2c927464303a8a661264435dc61" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 35851 } } peers { permanent_uuid: "1217932a671e4457bf8c008451fafd01" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 46753 } } peers { permanent_uuid: "12a76517a01541d0ab63360f1b9c04c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 40347 } }
I20260925 15:20:16.681293 5540 ts_tablet_manager.cc:1428] T fdba6b2348df4b09b5bef67363ce3af2 P 12a76517a01541d0ab63360f1b9c04c5: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 15:20:16.703464 5548 raft_consensus.cc:482] T fdba6b2348df4b09b5bef67363ce3af2 P aecdd2c927464303a8a661264435dc61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:20:16.703581 5548 raft_consensus.cc:504] T fdba6b2348df4b09b5bef67363ce3af2 P aecdd2c927464303a8a661264435dc61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aecdd2c927464303a8a661264435dc61" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 35851 } } peers { permanent_uuid: "1217932a671e4457bf8c008451fafd01" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 46753 } } peers { permanent_uuid: "12a76517a01541d0ab63360f1b9c04c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 40347 } }
I20260925 15:20:16.703739 5548 leader_election.cc:288] T fdba6b2348df4b09b5bef67363ce3af2 P aecdd2c927464303a8a661264435dc61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1217932a671e4457bf8c008451fafd01 (127.3.211.194:46753), 12a76517a01541d0ab63360f1b9c04c5 (127.3.211.195:40347)
I20260925 15:20:16.703985 5471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdba6b2348df4b09b5bef67363ce3af2" candidate_uuid: "aecdd2c927464303a8a661264435dc61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12a76517a01541d0ab63360f1b9c04c5" is_pre_election: true
I20260925 15:20:16.703985 5331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdba6b2348df4b09b5bef67363ce3af2" candidate_uuid: "aecdd2c927464303a8a661264435dc61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1217932a671e4457bf8c008451fafd01" is_pre_election: true
I20260925 15:20:16.704097 5471 raft_consensus.cc:2454] T fdba6b2348df4b09b5bef67363ce3af2 P 12a76517a01541d0ab63360f1b9c04c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aecdd2c927464303a8a661264435dc61 in term 0.
I20260925 15:20:16.704097 5331 raft_consensus.cc:2454] T fdba6b2348df4b09b5bef67363ce3af2 P 1217932a671e4457bf8c008451fafd01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aecdd2c927464303a8a661264435dc61 in term 0.
I20260925 15:20:16.704259 5124 leader_election.cc:302] T fdba6b2348df4b09b5bef67363ce3af2 P aecdd2c927464303a8a661264435dc61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12a76517a01541d0ab63360f1b9c04c5, aecdd2c927464303a8a661264435dc61; no voters:
I20260925 15:20:16.704398 5548 raft_consensus.cc:2781] T fdba6b2348df4b09b5bef67363ce3af2 P aecdd2c927464303a8a661264435dc61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:20:16.704495 5548 raft_consensus.cc:482] T fdba6b2348df4b09b5bef67363ce3af2 P aecdd2c927464303a8a661264435dc61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:20:16.704530 5548 raft_consensus.cc:3037] T fdba6b2348df4b09b5bef67363ce3af2 P aecdd2c927464303a8a661264435dc61 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:16.705470 5548 raft_consensus.cc:504] T fdba6b2348df4b09b5bef67363ce3af2 P aecdd2c927464303a8a661264435dc61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aecdd2c927464303a8a661264435dc61" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 35851 } } peers { permanent_uuid: "1217932a671e4457bf8c008451fafd01" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 46753 } } peers { permanent_uuid: "12a76517a01541d0ab63360f1b9c04c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 40347 } }
I20260925 15:20:16.705778 5548 leader_election.cc:288] T fdba6b2348df4b09b5bef67363ce3af2 P aecdd2c927464303a8a661264435dc61 [CANDIDATE]: Term 1 election: Requested vote from peers 1217932a671e4457bf8c008451fafd01 (127.3.211.194:46753), 12a76517a01541d0ab63360f1b9c04c5 (127.3.211.195:40347)
I20260925 15:20:16.706069 5471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdba6b2348df4b09b5bef67363ce3af2" candidate_uuid: "aecdd2c927464303a8a661264435dc61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12a76517a01541d0ab63360f1b9c04c5"
I20260925 15:20:16.706131 5331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdba6b2348df4b09b5bef67363ce3af2" candidate_uuid: "aecdd2c927464303a8a661264435dc61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1217932a671e4457bf8c008451fafd01"
I20260925 15:20:16.706213 5471 raft_consensus.cc:3037] T fdba6b2348df4b09b5bef67363ce3af2 P 12a76517a01541d0ab63360f1b9c04c5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:16.706236 5331 raft_consensus.cc:3037] T fdba6b2348df4b09b5bef67363ce3af2 P 1217932a671e4457bf8c008451fafd01 [term 0 FOLLOWER]: Advancing to term 1
W20260925 15:20:16.706408 5238 tablet.cc:2369] T fdba6b2348df4b09b5bef67363ce3af2 P aecdd2c927464303a8a661264435dc61: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:20:16.707247 5471 raft_consensus.cc:2454] T fdba6b2348df4b09b5bef67363ce3af2 P 12a76517a01541d0ab63360f1b9c04c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aecdd2c927464303a8a661264435dc61 in term 1.
I20260925 15:20:16.707538 5331 raft_consensus.cc:2454] T fdba6b2348df4b09b5bef67363ce3af2 P 1217932a671e4457bf8c008451fafd01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aecdd2c927464303a8a661264435dc61 in term 1.
I20260925 15:20:16.707554 5124 leader_election.cc:302] T fdba6b2348df4b09b5bef67363ce3af2 P aecdd2c927464303a8a661264435dc61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12a76517a01541d0ab63360f1b9c04c5, aecdd2c927464303a8a661264435dc61; no voters:
I20260925 15:20:16.707686 5548 raft_consensus.cc:2781] T fdba6b2348df4b09b5bef67363ce3af2 P aecdd2c927464303a8a661264435dc61 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:20:16.707749 5548 raft_consensus.cc:686] T fdba6b2348df4b09b5bef67363ce3af2 P aecdd2c927464303a8a661264435dc61 [term 1 LEADER]: Becoming Leader. State: Replica: aecdd2c927464303a8a661264435dc61, State: Running, Role: LEADER
I20260925 15:20:16.707818 5548 consensus_queue.cc:237] T fdba6b2348df4b09b5bef67363ce3af2 P aecdd2c927464303a8a661264435dc61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aecdd2c927464303a8a661264435dc61" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 35851 } } peers { permanent_uuid: "1217932a671e4457bf8c008451fafd01" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 46753 } } peers { permanent_uuid: "12a76517a01541d0ab63360f1b9c04c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 40347 } }
I20260925 15:20:16.708725 5034 catalog_manager.cc:5495] T fdba6b2348df4b09b5bef67363ce3af2 P aecdd2c927464303a8a661264435dc61 reported cstate change: term changed from 0 to 1, leader changed from <none> to aecdd2c927464303a8a661264435dc61 (127.3.211.193). New cstate: current_term: 1 leader_uuid: "aecdd2c927464303a8a661264435dc61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aecdd2c927464303a8a661264435dc61" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 35851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1217932a671e4457bf8c008451fafd01" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 46753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12a76517a01541d0ab63360f1b9c04c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 40347 } health_report { overall_health: UNKNOWN } } }
Sep 25 15:20:16 dist-test-slave-4bk3 krb5kdc[4989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349615, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.254@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:20:16.740922 5083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:16.733238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53554 (local address 127.3.211.254:40471)
0925 15:20:16.733313 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:16.733316 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:16.733433 (+ 117us) server_negotiation.cc:407] Connection header received
0925 15:20:16.733506 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:16.733506 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:16.733517 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:16.733590 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:16.734066 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.734564 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.735283 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.735461 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.735550 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:16.738018 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:16.738019 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:16.738020 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:16.738027 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:16.739443 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:16.739950 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:16.739951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:16.739952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:16.740019 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:16.740246 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:16.740247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:16.740247 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:16.740430 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:16.740522 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:16.740697 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 15:20:16.740807 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 15:20:16.740998 5576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:16.733182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:40471 (local address 127.0.0.1:53554)
0925 15:20:16.733351 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:16.733361 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:16.733453 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:16.733592 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:16.733593 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:16.733922 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:16.733968 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:16.733971 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.734667 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.734667 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:16.735182 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:16.735183 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.735563 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.735564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:16.735623 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:16.736241 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:16.736246 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:16.737912 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:16.739567 (+ 1655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:16.739577 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:16.739580 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:16.739793 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:16.740109 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:16.740110 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:16.740110 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:16.740175 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:16.740509 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:16.740518 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:16.740566 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 15:20:16.740671 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 15:20:16.740869 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":83,"thread_start_us":97,"threads_started":1}
I20260925 15:20:16.744011 5033 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:53554:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 15:20:16.744230 5033 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:20:16.745774 5311 tablet_service.cc:1467] Processing CreateTablet for tablet 8ad738d3f7a94f308ae3c1dbe793ba1b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5b76afc495a2436d84801daf437a772e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:20:16.745788 5171 tablet_service.cc:1467] Processing CreateTablet for tablet 8ad738d3f7a94f308ae3c1dbe793ba1b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5b76afc495a2436d84801daf437a772e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:20:16.745975 5171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ad738d3f7a94f308ae3c1dbe793ba1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:16.745975 5311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ad738d3f7a94f308ae3c1dbe793ba1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:16.747256 5541 tablet_bootstrap.cc:492] T 8ad738d3f7a94f308ae3c1dbe793ba1b P 1217932a671e4457bf8c008451fafd01: Bootstrap starting.
I20260925 15:20:16.747536 5542 tablet_bootstrap.cc:492] T 8ad738d3f7a94f308ae3c1dbe793ba1b P aecdd2c927464303a8a661264435dc61: Bootstrap starting.
I20260925 15:20:16.747658 5451 tablet_service.cc:1467] Processing CreateTablet for tablet 8ad738d3f7a94f308ae3c1dbe793ba1b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5b76afc495a2436d84801daf437a772e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:20:16.747735 5541 tablet_bootstrap.cc:654] T 8ad738d3f7a94f308ae3c1dbe793ba1b P 1217932a671e4457bf8c008451fafd01: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:16.747815 5451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ad738d3f7a94f308ae3c1dbe793ba1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:16.747991 5542 tablet_bootstrap.cc:654] T 8ad738d3f7a94f308ae3c1dbe793ba1b P aecdd2c927464303a8a661264435dc61: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:16.748705 5542 tablet_bootstrap.cc:492] T 8ad738d3f7a94f308ae3c1dbe793ba1b P aecdd2c927464303a8a661264435dc61: No bootstrap required, opened a new log
I20260925 15:20:16.748775 5542 ts_tablet_manager.cc:1397] T 8ad738d3f7a94f308ae3c1dbe793ba1b P aecdd2c927464303a8a661264435dc61: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:16.748929 5542 raft_consensus.cc:348] T 8ad738d3f7a94f308ae3c1dbe793ba1b P aecdd2c927464303a8a661264435dc61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aecdd2c927464303a8a661264435dc61" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 35851 } } peers { permanent_uuid: "1217932a671e4457bf8c008451fafd01" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 46753 } } peers { permanent_uuid: "12a76517a01541d0ab63360f1b9c04c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 40347 } }
I20260925 15:20:16.749010 5542 raft_consensus.cc:374] T 8ad738d3f7a94f308ae3c1dbe793ba1b P aecdd2c927464303a8a661264435dc61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:16.749038 5542 raft_consensus.cc:729] T 8ad738d3f7a94f308ae3c1dbe793ba1b P aecdd2c927464303a8a661264435dc61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aecdd2c927464303a8a661264435dc61, State: Initialized, Role: FOLLOWER
I20260925 15:20:16.749094 5542 consensus_queue.cc:260] T 8ad738d3f7a94f308ae3c1dbe793ba1b P aecdd2c927464303a8a661264435dc61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aecdd2c927464303a8a661264435dc61" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 35851 } } peers { permanent_uuid: "1217932a671e4457bf8c008451fafd01" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 46753 } } peers { permanent_uuid: "12a76517a01541d0ab63360f1b9c04c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 40347 } }
I20260925 15:20:16.749194 5542 ts_tablet_manager.cc:1428] T 8ad738d3f7a94f308ae3c1dbe793ba1b P aecdd2c927464303a8a661264435dc61: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 15:20:16.749281 5548 tablet_replica.cc:439] T 8ad738d3f7a94f308ae3c1dbe793ba1b P aecdd2c927464303a8a661264435dc61: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aecdd2c927464303a8a661264435dc61" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 35851 } } peers { permanent_uuid: "1217932a671e4457bf8c008451fafd01" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 46753 } } peers { permanent_uuid: "12a76517a01541d0ab63360f1b9c04c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 40347 } } }
I20260925 15:20:16.749364 5548 tablet_replica.cc:442] T 8ad738d3f7a94f308ae3c1dbe793ba1b P aecdd2c927464303a8a661264435dc61: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:20:16.749660 5540 tablet_bootstrap.cc:492] T 8ad738d3f7a94f308ae3c1dbe793ba1b P 12a76517a01541d0ab63360f1b9c04c5: Bootstrap starting.
I20260925 15:20:16.750088 5540 tablet_bootstrap.cc:654] T 8ad738d3f7a94f308ae3c1dbe793ba1b P 12a76517a01541d0ab63360f1b9c04c5: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:16.750109 5541 tablet_bootstrap.cc:492] T 8ad738d3f7a94f308ae3c1dbe793ba1b P 1217932a671e4457bf8c008451fafd01: No bootstrap required, opened a new log
I20260925 15:20:16.750168 5541 ts_tablet_manager.cc:1397] T 8ad738d3f7a94f308ae3c1dbe793ba1b P 1217932a671e4457bf8c008451fafd01: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260925 15:20:16.750509 5541 raft_consensus.cc:348] T 8ad738d3f7a94f308ae3c1dbe793ba1b P 1217932a671e4457bf8c008451fafd01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aecdd2c927464303a8a661264435dc61" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 35851 } } peers { permanent_uuid: "1217932a671e4457bf8c008451fafd01" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 46753 } } peers { permanent_uuid: "12a76517a01541d0ab63360f1b9c04c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 40347 } }
I20260925 15:20:16.750598 5541 raft_consensus.cc:374] T 8ad738d3f7a94f308ae3c1dbe793ba1b P 1217932a671e4457bf8c008451fafd01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:16.750635 5541 raft_consensus.cc:729] T 8ad738d3f7a94f308ae3c1dbe793ba1b P 1217932a671e4457bf8c008451fafd01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1217932a671e4457bf8c008451fafd01, State: Initialized, Role: FOLLOWER
I20260925 15:20:16.750679 5541 consensus_queue.cc:260] T 8ad738d3f7a94f308ae3c1dbe793ba1b P 1217932a671e4457bf8c008451fafd01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aecdd2c927464303a8a661264435dc61" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 35851 } } peers { permanent_uuid: "1217932a671e4457bf8c008451fafd01" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 46753 } } peers { permanent_uuid: "12a76517a01541d0ab63360f1b9c04c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 40347 } }
I20260925 15:20:16.750778 5540 tablet_bootstrap.cc:492] T 8ad738d3f7a94f308ae3c1dbe793ba1b P 12a76517a01541d0ab63360f1b9c04c5: No bootstrap required, opened a new log
I20260925 15:20:16.750795 5541 ts_tablet_manager.cc:1428] T 8ad738d3f7a94f308ae3c1dbe793ba1b P 1217932a671e4457bf8c008451fafd01: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260925 15:20:16.750835 5540 ts_tablet_manager.cc:1397] T 8ad738d3f7a94f308ae3c1dbe793ba1b P 12a76517a01541d0ab63360f1b9c04c5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:16.750938 5547 tablet_replica.cc:439] T 8ad738d3f7a94f308ae3c1dbe793ba1b P 1217932a671e4457bf8c008451fafd01: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aecdd2c927464303a8a661264435dc61" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 35851 } } peers { permanent_uuid: "1217932a671e4457bf8c008451fafd01" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 46753 } } peers { permanent_uuid: "12a76517a01541d0ab63360f1b9c04c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 40347 } } }
I20260925 15:20:16.750990 5540 raft_consensus.cc:348] T 8ad738d3f7a94f308ae3c1dbe793ba1b P 12a76517a01541d0ab63360f1b9c04c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aecdd2c927464303a8a661264435dc61" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 35851 } } peers { permanent_uuid: "1217932a671e4457bf8c008451fafd01" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 46753 } } peers { permanent_uuid: "12a76517a01541d0ab63360f1b9c04c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 40347 } }
I20260925 15:20:16.751027 5547 tablet_replica.cc:442] T 8ad738d3f7a94f308ae3c1dbe793ba1b P 1217932a671e4457bf8c008451fafd01: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:20:16.751078 5540 raft_consensus.cc:374] T 8ad738d3f7a94f308ae3c1dbe793ba1b P 12a76517a01541d0ab63360f1b9c04c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:16.751112 5540 raft_consensus.cc:729] T 8ad738d3f7a94f308ae3c1dbe793ba1b P 12a76517a01541d0ab63360f1b9c04c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12a76517a01541d0ab63360f1b9c04c5, State: Initialized, Role: FOLLOWER
I20260925 15:20:16.751155 5540 consensus_queue.cc:260] T 8ad738d3f7a94f308ae3c1dbe793ba1b P 12a76517a01541d0ab63360f1b9c04c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aecdd2c927464303a8a661264435dc61" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 35851 } } peers { permanent_uuid: "1217932a671e4457bf8c008451fafd01" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 46753 } } peers { permanent_uuid: "12a76517a01541d0ab63360f1b9c04c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 40347 } }
I20260925 15:20:16.751266 5540 ts_tablet_manager.cc:1428] T 8ad738d3f7a94f308ae3c1dbe793ba1b P 12a76517a01541d0ab63360f1b9c04c5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:16.751324 5546 tablet_replica.cc:439] T 8ad738d3f7a94f308ae3c1dbe793ba1b P 12a76517a01541d0ab63360f1b9c04c5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aecdd2c927464303a8a661264435dc61" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 35851 } } peers { permanent_uuid: "1217932a671e4457bf8c008451fafd01" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 46753 } } peers { permanent_uuid: "12a76517a01541d0ab63360f1b9c04c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 40347 } } }
I20260925 15:20:16.751427 5546 tablet_replica.cc:442] T 8ad738d3f7a94f308ae3c1dbe793ba1b P 12a76517a01541d0ab63360f1b9c04c5: This TxnStatusTablet replica's current role is: FOLLOWER
W20260925 15:20:16.793215 5518 tablet.cc:2369] T 8ad738d3f7a94f308ae3c1dbe793ba1b P 12a76517a01541d0ab63360f1b9c04c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:20:16.903375 5548 raft_consensus.cc:482] T 8ad738d3f7a94f308ae3c1dbe793ba1b P aecdd2c927464303a8a661264435dc61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:20:16.903483 5548 raft_consensus.cc:504] T 8ad738d3f7a94f308ae3c1dbe793ba1b P aecdd2c927464303a8a661264435dc61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aecdd2c927464303a8a661264435dc61" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 35851 } } peers { permanent_uuid: "1217932a671e4457bf8c008451fafd01" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 46753 } } peers { permanent_uuid: "12a76517a01541d0ab63360f1b9c04c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 40347 } }
I20260925 15:20:16.903702 5548 leader_election.cc:288] T 8ad738d3f7a94f308ae3c1dbe793ba1b P aecdd2c927464303a8a661264435dc61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1217932a671e4457bf8c008451fafd01 (127.3.211.194:46753), 12a76517a01541d0ab63360f1b9c04c5 (127.3.211.195:40347)
I20260925 15:20:16.903957 5471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ad738d3f7a94f308ae3c1dbe793ba1b" candidate_uuid: "aecdd2c927464303a8a661264435dc61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12a76517a01541d0ab63360f1b9c04c5" is_pre_election: true
I20260925 15:20:16.903975 5331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ad738d3f7a94f308ae3c1dbe793ba1b" candidate_uuid: "aecdd2c927464303a8a661264435dc61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1217932a671e4457bf8c008451fafd01" is_pre_election: true
I20260925 15:20:16.904072 5471 raft_consensus.cc:2454] T 8ad738d3f7a94f308ae3c1dbe793ba1b P 12a76517a01541d0ab63360f1b9c04c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aecdd2c927464303a8a661264435dc61 in term 0.
I20260925 15:20:16.904070 5331 raft_consensus.cc:2454] T 8ad738d3f7a94f308ae3c1dbe793ba1b P 1217932a671e4457bf8c008451fafd01 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aecdd2c927464303a8a661264435dc61 in term 0.
I20260925 15:20:16.904240 5125 leader_election.cc:302] T 8ad738d3f7a94f308ae3c1dbe793ba1b P aecdd2c927464303a8a661264435dc61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1217932a671e4457bf8c008451fafd01, aecdd2c927464303a8a661264435dc61; no voters:
I20260925 15:20:16.904382 5548 raft_consensus.cc:2781] T 8ad738d3f7a94f308ae3c1dbe793ba1b P aecdd2c927464303a8a661264435dc61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:20:16.904423 5548 raft_consensus.cc:482] T 8ad738d3f7a94f308ae3c1dbe793ba1b P aecdd2c927464303a8a661264435dc61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:20:16.904449 5548 raft_consensus.cc:3037] T 8ad738d3f7a94f308ae3c1dbe793ba1b P aecdd2c927464303a8a661264435dc61 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:16.905249 5548 raft_consensus.cc:504] T 8ad738d3f7a94f308ae3c1dbe793ba1b P aecdd2c927464303a8a661264435dc61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aecdd2c927464303a8a661264435dc61" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 35851 } } peers { permanent_uuid: "1217932a671e4457bf8c008451fafd01" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 46753 } } peers { permanent_uuid: "12a76517a01541d0ab63360f1b9c04c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 40347 } }
I20260925 15:20:16.905421 5548 leader_election.cc:288] T 8ad738d3f7a94f308ae3c1dbe793ba1b P aecdd2c927464303a8a661264435dc61 [CANDIDATE]: Term 1 election: Requested vote from peers 1217932a671e4457bf8c008451fafd01 (127.3.211.194:46753), 12a76517a01541d0ab63360f1b9c04c5 (127.3.211.195:40347)
I20260925 15:20:16.905561 5331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ad738d3f7a94f308ae3c1dbe793ba1b" candidate_uuid: "aecdd2c927464303a8a661264435dc61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1217932a671e4457bf8c008451fafd01"
I20260925 15:20:16.905663 5331 raft_consensus.cc:3037] T 8ad738d3f7a94f308ae3c1dbe793ba1b P 1217932a671e4457bf8c008451fafd01 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:16.905638 5471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ad738d3f7a94f308ae3c1dbe793ba1b" candidate_uuid: "aecdd2c927464303a8a661264435dc61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12a76517a01541d0ab63360f1b9c04c5"
I20260925 15:20:16.905745 5471 raft_consensus.cc:3037] T 8ad738d3f7a94f308ae3c1dbe793ba1b P 12a76517a01541d0ab63360f1b9c04c5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:16.906486 5471 raft_consensus.cc:2454] T 8ad738d3f7a94f308ae3c1dbe793ba1b P 12a76517a01541d0ab63360f1b9c04c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aecdd2c927464303a8a661264435dc61 in term 1.
I20260925 15:20:16.906538 5331 raft_consensus.cc:2454] T 8ad738d3f7a94f308ae3c1dbe793ba1b P 1217932a671e4457bf8c008451fafd01 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aecdd2c927464303a8a661264435dc61 in term 1.
I20260925 15:20:16.906723 5124 leader_election.cc:302] T 8ad738d3f7a94f308ae3c1dbe793ba1b P aecdd2c927464303a8a661264435dc61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12a76517a01541d0ab63360f1b9c04c5, aecdd2c927464303a8a661264435dc61; no voters:
I20260925 15:20:16.906836 5548 raft_consensus.cc:2781] T 8ad738d3f7a94f308ae3c1dbe793ba1b P aecdd2c927464303a8a661264435dc61 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:20:16.906909 5548 raft_consensus.cc:686] T 8ad738d3f7a94f308ae3c1dbe793ba1b P aecdd2c927464303a8a661264435dc61 [term 1 LEADER]: Becoming Leader. State: Replica: aecdd2c927464303a8a661264435dc61, State: Running, Role: LEADER
I20260925 15:20:16.906965 5548 consensus_queue.cc:237] T 8ad738d3f7a94f308ae3c1dbe793ba1b P aecdd2c927464303a8a661264435dc61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aecdd2c927464303a8a661264435dc61" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 35851 } } peers { permanent_uuid: "1217932a671e4457bf8c008451fafd01" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 46753 } } peers { permanent_uuid: "12a76517a01541d0ab63360f1b9c04c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 40347 } }
I20260925 15:20:16.907241 5552 tablet_replica.cc:439] T 8ad738d3f7a94f308ae3c1dbe793ba1b P aecdd2c927464303a8a661264435dc61: TxnStatusTablet state changed. Reason: New leader aecdd2c927464303a8a661264435dc61. Latest consensus state: current_term: 1 leader_uuid: "aecdd2c927464303a8a661264435dc61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aecdd2c927464303a8a661264435dc61" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 35851 } } peers { permanent_uuid: "1217932a671e4457bf8c008451fafd01" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 46753 } } peers { permanent_uuid: "12a76517a01541d0ab63360f1b9c04c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 40347 } } }
I20260925 15:20:16.907336 5552 tablet_replica.cc:442] T 8ad738d3f7a94f308ae3c1dbe793ba1b P aecdd2c927464303a8a661264435dc61: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:20:16.907478 5581 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 15:20:16.907594 5581 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 15:20:16.907657 5034 catalog_manager.cc:5495] T 8ad738d3f7a94f308ae3c1dbe793ba1b P aecdd2c927464303a8a661264435dc61 reported cstate change: term changed from 0 to 1, leader changed from <none> to aecdd2c927464303a8a661264435dc61 (127.3.211.193). New cstate: current_term: 1 leader_uuid: "aecdd2c927464303a8a661264435dc61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aecdd2c927464303a8a661264435dc61" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 35851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1217932a671e4457bf8c008451fafd01" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 46753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12a76517a01541d0ab63360f1b9c04c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 40347 } health_report { overall_health: UNKNOWN } } }
I20260925 15:20:16.936962 5576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:16.934388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:35851 (local address 127.0.0.1:55880)
0925 15:20:16.934465 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:16.934476 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:16.934588 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:16.934766 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:16.934767 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:16.934769 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:20:16.934823 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:16.934826 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.935483 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.935485 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:16.936056 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:16.936057 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.936489 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.936490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:16.936568 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:16.936585 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:20:16.936790 (+ 205us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:20:16.936797 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 15:20:16.936842 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 15:20:16.936894 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 15:20:16.936954 5538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:16.934471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55880 (local address 127.3.211.193:35851)
0925 15:20:16.934536 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:16.934538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:16.934565 (+ 27us) server_negotiation.cc:407] Connection header received
0925 15:20:16.934673 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:16.934675 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:16.934685 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:16.934758 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:20:16.934899 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.935384 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.936149 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.936401 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.936487 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:16.936682 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:20:16.936720 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:20:16.936772 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:16.936844 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 15:20:16.936868 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 15:20:16.938126 5331 raft_consensus.cc:1261] T 8ad738d3f7a94f308ae3c1dbe793ba1b P 1217932a671e4457bf8c008451fafd01 [term 1 FOLLOWER]: Refusing update from remote peer aecdd2c927464303a8a661264435dc61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:20:16.938125 5471 raft_consensus.cc:1261] T 8ad738d3f7a94f308ae3c1dbe793ba1b P 12a76517a01541d0ab63360f1b9c04c5 [term 1 FOLLOWER]: Refusing update from remote peer aecdd2c927464303a8a661264435dc61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:20:16.938354 5552 consensus_queue.cc:1035] T 8ad738d3f7a94f308ae3c1dbe793ba1b P aecdd2c927464303a8a661264435dc61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1217932a671e4457bf8c008451fafd01" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 46753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:20:16.938483 5552 consensus_queue.cc:1035] T 8ad738d3f7a94f308ae3c1dbe793ba1b P aecdd2c927464303a8a661264435dc61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12a76517a01541d0ab63360f1b9c04c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 40347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:20:16.938895 5547 tablet_replica.cc:439] T 8ad738d3f7a94f308ae3c1dbe793ba1b P 1217932a671e4457bf8c008451fafd01: TxnStatusTablet state changed. Reason: New leader aecdd2c927464303a8a661264435dc61. Latest consensus state: current_term: 1 leader_uuid: "aecdd2c927464303a8a661264435dc61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aecdd2c927464303a8a661264435dc61" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 35851 } } peers { permanent_uuid: "1217932a671e4457bf8c008451fafd01" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 46753 } } peers { permanent_uuid: "12a76517a01541d0ab63360f1b9c04c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 40347 } } }
I20260925 15:20:16.938993 5547 tablet_replica.cc:442] T 8ad738d3f7a94f308ae3c1dbe793ba1b P 1217932a671e4457bf8c008451fafd01: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:20:16.938951 5546 tablet_replica.cc:439] T 8ad738d3f7a94f308ae3c1dbe793ba1b P 12a76517a01541d0ab63360f1b9c04c5: TxnStatusTablet state changed. Reason: New leader aecdd2c927464303a8a661264435dc61. Latest consensus state: current_term: 1 leader_uuid: "aecdd2c927464303a8a661264435dc61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aecdd2c927464303a8a661264435dc61" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 35851 } } peers { permanent_uuid: "1217932a671e4457bf8c008451fafd01" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 46753 } } peers { permanent_uuid: "12a76517a01541d0ab63360f1b9c04c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 40347 } } }
I20260925 15:20:16.939033 5546 tablet_replica.cc:442] T 8ad738d3f7a94f308ae3c1dbe793ba1b P 12a76517a01541d0ab63360f1b9c04c5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:20:16.939736 5552 tablet_replica.cc:439] T 8ad738d3f7a94f308ae3c1dbe793ba1b P aecdd2c927464303a8a661264435dc61: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aecdd2c927464303a8a661264435dc61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aecdd2c927464303a8a661264435dc61" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 35851 } } peers { permanent_uuid: "1217932a671e4457bf8c008451fafd01" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 46753 } } peers { permanent_uuid: "12a76517a01541d0ab63360f1b9c04c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 40347 } } }
I20260925 15:20:16.939827 5552 tablet_replica.cc:442] T 8ad738d3f7a94f308ae3c1dbe793ba1b P aecdd2c927464303a8a661264435dc61: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:20:16.940172 5547 tablet_replica.cc:439] T 8ad738d3f7a94f308ae3c1dbe793ba1b P 1217932a671e4457bf8c008451fafd01: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aecdd2c927464303a8a661264435dc61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aecdd2c927464303a8a661264435dc61" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 35851 } } peers { permanent_uuid: "1217932a671e4457bf8c008451fafd01" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 46753 } } peers { permanent_uuid: "12a76517a01541d0ab63360f1b9c04c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 40347 } } }
I20260925 15:20:16.940258 5547 tablet_replica.cc:442] T 8ad738d3f7a94f308ae3c1dbe793ba1b P 1217932a671e4457bf8c008451fafd01: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:20:16.941107 5552 tablet_replica.cc:439] T 8ad738d3f7a94f308ae3c1dbe793ba1b P aecdd2c927464303a8a661264435dc61: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aecdd2c927464303a8a661264435dc61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aecdd2c927464303a8a661264435dc61" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 35851 } } peers { permanent_uuid: "1217932a671e4457bf8c008451fafd01" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 46753 } } peers { permanent_uuid: "12a76517a01541d0ab63360f1b9c04c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 40347 } } }
I20260925 15:20:16.941196 5552 tablet_replica.cc:442] T 8ad738d3f7a94f308ae3c1dbe793ba1b P aecdd2c927464303a8a661264435dc61: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:20:16.941478 5546 tablet_replica.cc:439] T 8ad738d3f7a94f308ae3c1dbe793ba1b P 12a76517a01541d0ab63360f1b9c04c5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aecdd2c927464303a8a661264435dc61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aecdd2c927464303a8a661264435dc61" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 35851 } } peers { permanent_uuid: "1217932a671e4457bf8c008451fafd01" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 46753 } } peers { permanent_uuid: "12a76517a01541d0ab63360f1b9c04c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 40347 } } }
I20260925 15:20:16.941562 5546 tablet_replica.cc:442] T 8ad738d3f7a94f308ae3c1dbe793ba1b P 12a76517a01541d0ab63360f1b9c04c5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:20:16.946449 5538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:16.943564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:33305 (local address 127.3.211.193:35851)
0925 15:20:16.943629 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:16.943631 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:16.943711 (+ 80us) server_negotiation.cc:407] Connection header received
0925 15:20:16.943785 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:16.943786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:16.943794 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:16.943832 (+ 38us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:20:16.944010 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.944638 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.945744 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.945976 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.946038 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:16.946211 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:20:16.946237 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:20:16.946276 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:16.946352 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 15:20:16.946390 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":15,"server-negotiator.queue_time_us":45}
I20260925 15:20:16.946561 5588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:16.943458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:35851 (local address 127.3.211.193:33305)
0925 15:20:16.943653 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:16.943665 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:16.943728 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:16.943845 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:16.943845 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:16.943847 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:20:16.943898 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:16.943899 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.944733 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.944734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:16.945650 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:16.945652 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.946052 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.946053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:16.946109 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:16.946129 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:20:16.946293 (+ 164us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:20:16.946295 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 15:20:16.946350 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 15:20:16.946429 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":2,"thread_start_us":121,"threads_started":1}
I20260925 15:20:16.949005 5471 raft_consensus.cc:1261] T fdba6b2348df4b09b5bef67363ce3af2 P 12a76517a01541d0ab63360f1b9c04c5 [term 1 FOLLOWER]: Refusing update from remote peer aecdd2c927464303a8a661264435dc61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:20:16.949028 5331 raft_consensus.cc:1261] T fdba6b2348df4b09b5bef67363ce3af2 P 1217932a671e4457bf8c008451fafd01 [term 1 FOLLOWER]: Refusing update from remote peer aecdd2c927464303a8a661264435dc61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:20:16.949260 5552 consensus_queue.cc:1035] T fdba6b2348df4b09b5bef67363ce3af2 P aecdd2c927464303a8a661264435dc61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12a76517a01541d0ab63360f1b9c04c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 40347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:20:16.949414 5552 consensus_queue.cc:1035] T fdba6b2348df4b09b5bef67363ce3af2 P aecdd2c927464303a8a661264435dc61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1217932a671e4457bf8c008451fafd01" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 46753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:20:16.964398 5033 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:20:16.964532 5033 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53546:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:20:16.965488 5033 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 980593882e994988bb22962652e15a9d: Sending DeleteTablet for 3 replicas of tablet fdba6b2348df4b09b5bef67363ce3af2
I20260925 15:20:16.965813 5311 tablet_service.cc:1514] Processing DeleteTablet for tablet fdba6b2348df4b09b5bef67363ce3af2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:20:16 UTC) from {username='kudu', principal='kudu/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:57084
I20260925 15:20:16.965816 5171 tablet_service.cc:1514] Processing DeleteTablet for tablet fdba6b2348df4b09b5bef67363ce3af2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:20:16 UTC) from {username='kudu', principal='kudu/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:55864
I20260925 15:20:16.965922 5566 tablet_replica.cc:330] T fdba6b2348df4b09b5bef67363ce3af2 P 1217932a671e4457bf8c008451fafd01: stopping tablet replica
I20260925 15:20:16.965976 5566 raft_consensus.cc:2229] T fdba6b2348df4b09b5bef67363ce3af2 P 1217932a671e4457bf8c008451fafd01 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:20:16.966027 5566 raft_consensus.cc:2258] T fdba6b2348df4b09b5bef67363ce3af2 P 1217932a671e4457bf8c008451fafd01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:20:16.966095 5565 tablet_replica.cc:330] T fdba6b2348df4b09b5bef67363ce3af2 P aecdd2c927464303a8a661264435dc61: stopping tablet replica
I20260925 15:20:16.966141 5565 raft_consensus.cc:2229] T fdba6b2348df4b09b5bef67363ce3af2 P aecdd2c927464303a8a661264435dc61 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:20:16.966192 5565 raft_consensus.cc:2258] T fdba6b2348df4b09b5bef67363ce3af2 P aecdd2c927464303a8a661264435dc61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:20:16.966497 5565 ts_tablet_manager.cc:1905] T fdba6b2348df4b09b5bef67363ce3af2 P aecdd2c927464303a8a661264435dc61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:20:16.968269 5565 ts_tablet_manager.cc:1918] T fdba6b2348df4b09b5bef67363ce3af2 P aecdd2c927464303a8a661264435dc61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:20:16.968369 5565 log.cc:1196] T fdba6b2348df4b09b5bef67363ce3af2 P aecdd2c927464303a8a661264435dc61: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/ts-0/wal/wals/fdba6b2348df4b09b5bef67363ce3af2
I20260925 15:20:16.968322 5451 tablet_service.cc:1514] Processing DeleteTablet for tablet fdba6b2348df4b09b5bef67363ce3af2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:20:16 UTC) from {username='kudu', principal='kudu/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:42020
I20260925 15:20:16.968446 5564 tablet_replica.cc:330] T fdba6b2348df4b09b5bef67363ce3af2 P 12a76517a01541d0ab63360f1b9c04c5: stopping tablet replica
I20260925 15:20:16.968489 5566 ts_tablet_manager.cc:1905] T fdba6b2348df4b09b5bef67363ce3af2 P 1217932a671e4457bf8c008451fafd01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:20:16.968528 5564 raft_consensus.cc:2229] T fdba6b2348df4b09b5bef67363ce3af2 P 12a76517a01541d0ab63360f1b9c04c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:20:16.968572 5564 raft_consensus.cc:2258] T fdba6b2348df4b09b5bef67363ce3af2 P 12a76517a01541d0ab63360f1b9c04c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:20:16.968639 5565 ts_tablet_manager.cc:1939] T fdba6b2348df4b09b5bef67363ce3af2 P aecdd2c927464303a8a661264435dc61: Deleting consensus metadata
I20260925 15:20:16.969225 5019 catalog_manager.cc:4841] TS aecdd2c927464303a8a661264435dc61 (127.3.211.193:35851): tablet fdba6b2348df4b09b5bef67363ce3af2 (table test-table [id=dfa89c7cad1a4af4bf2ef208d73d1e99]) successfully deleted
I20260925 15:20:16.969537 5564 ts_tablet_manager.cc:1905] T fdba6b2348df4b09b5bef67363ce3af2 P 12a76517a01541d0ab63360f1b9c04c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:20:16.970055 5566 ts_tablet_manager.cc:1918] T fdba6b2348df4b09b5bef67363ce3af2 P 1217932a671e4457bf8c008451fafd01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:20:16.970110 5566 log.cc:1196] T fdba6b2348df4b09b5bef67363ce3af2 P 1217932a671e4457bf8c008451fafd01: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/ts-1/wal/wals/fdba6b2348df4b09b5bef67363ce3af2
I20260925 15:20:16.970350 5566 ts_tablet_manager.cc:1939] T fdba6b2348df4b09b5bef67363ce3af2 P 1217932a671e4457bf8c008451fafd01: Deleting consensus metadata
I20260925 15:20:16.971001 5021 catalog_manager.cc:4841] TS 1217932a671e4457bf8c008451fafd01 (127.3.211.194:46753): tablet fdba6b2348df4b09b5bef67363ce3af2 (table test-table [id=dfa89c7cad1a4af4bf2ef208d73d1e99]) successfully deleted
I20260925 15:20:16.971391 5564 ts_tablet_manager.cc:1918] T fdba6b2348df4b09b5bef67363ce3af2 P 12a76517a01541d0ab63360f1b9c04c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:20:16.971446 5564 log.cc:1196] T fdba6b2348df4b09b5bef67363ce3af2 P 12a76517a01541d0ab63360f1b9c04c5: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790349610595785-3919-0/minicluster-data/ts-2/wal/wals/fdba6b2348df4b09b5bef67363ce3af2
I20260925 15:20:16.971673 5564 ts_tablet_manager.cc:1939] T fdba6b2348df4b09b5bef67363ce3af2 P 12a76517a01541d0ab63360f1b9c04c5: Deleting consensus metadata
I20260925 15:20:16.971992 5020 catalog_manager.cc:4841] TS 12a76517a01541d0ab63360f1b9c04c5 (127.3.211.195:40347): tablet fdba6b2348df4b09b5bef67363ce3af2 (table test-table [id=dfa89c7cad1a4af4bf2ef208d73d1e99]) successfully deleted
Sep 25 15:20:16 dist-test-slave-4bk3 krb5kdc[4989](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349616, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.211.193@KRBTEST.COM
I20260925 15:20:16.983453 5538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:16.969378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55882 (local address 127.3.211.193:35851)
0925 15:20:16.969418 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:16.969421 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:16.969678 (+ 257us) server_negotiation.cc:407] Connection header received
0925 15:20:16.969781 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:16.969782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:16.969793 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:16.969823 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:16.970642 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.971367 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.972540 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.972845 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.972914 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:16.979814 (+ 6900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:16.979829 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:16.979829 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:16.979840 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:16.982009 (+ 2169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:16.982464 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:16.982469 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:16.982469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:16.982550 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:16.982797 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:16.982798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:16.982798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:16.982991 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:16.983150 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:16.983245 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 15:20:16.983330 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260925 15:20:16.983796 5231 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55882
I20260925 15:20:16.992014 5083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:16.984907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53560 (local address 127.3.211.254:40471)
0925 15:20:16.984953 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:16.984956 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:16.984983 (+ 27us) server_negotiation.cc:407] Connection header received
0925 15:20:16.985063 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:16.985064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:16.985072 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:16.985159 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:16.985807 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.986333 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.987307 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:16.987494 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:16.987556 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:16.989047 (+ 1491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:16.989049 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:16.989049 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:16.989064 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:16.990515 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:16.990938 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:16.990939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:16.990939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:16.991007 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:16.991329 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:16.991332 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:16.991333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:16.991545 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:16.991788 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:16.991857 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 15:20:16.991940 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260925 15:20:16.992651 5033 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53560
I20260925 15:20:16.993597 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 5106
I20260925 15:20:17.000875 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 5246
I20260925 15:20:17.005779 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 5386
I20260925 15:20:17.011276 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 5002
2026-09-25T15:20:17Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2362 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:20:17 dist-test-slave-4bk3 krb5kdc[5599](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:20:17 dist-test-slave-4bk3 krb5kdc[5599](info): set up 2 sockets
Sep 25 15:20:17 dist-test-slave-4bk3 krb5kdc[5599](info): commencing operation
krb5kdc: starting...
W20260925 15:20:18.076576 3919 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Sep 25 15:20:18 dist-test-slave-4bk3 krb5kdc[5599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:20:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:20:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.211.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
I20260925 15:20:18.203825 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:46877
--webserver_interface=127.3.211.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:34273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.211.254:46877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:20:18.291905 5612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:18.292143 5612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:18.292176 5612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:18.293488 5612 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:20:18.293541 5612 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 15:20:18.293555 5612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:18.293568 5612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:18.293581 5612 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:20:18.293594 5612 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:20:18.295218 5612 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:34273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.211.254:46877
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:46877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.211.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:18.295511 5612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:18.295785 5612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:18.300956 5618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:18.300992 5620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:18.300987 5617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:18.301326 5612 server_base.cc:1034] running on GCE node
I20260925 15:20:18.301596 5612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:18.301867 5612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:18.303123 5612 hybrid_clock.cc:648] HybridClock initialized: now 1790349618303092 us; error 45 us; skew 500 ppm
Sep 25 15:20:18 dist-test-slave-4bk3 krb5kdc[5599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349618, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:18.306738 5612 init.cc:377] Logged in from keytab as kudu/127.3.211.254@KRBTEST.COM (short username kudu)
I20260925 15:20:18.307245 5612 webserver.cc:460] Webserver started at http://127.3.211.254:44757/ using document root <none> and password file <none>
I20260925 15:20:18.307452 5612 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:18.307494 5612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:18.307585 5612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:18.308728 5612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/master-0/data/instance:
uuid: "bba60aadb5c245e6bb6d4680cdfa0adc"
format_stamp: "Formatted at 2026-09-25 15:20:18 on dist-test-slave-4bk3"
server_key: "07fc9eef175733eeac3e24e01b383947"
server_key_iv: "1b8a2e8bbcbdcd4848d9e8f74e103a9b"
server_key_version: "encryptionkey@0"
I20260925 15:20:18.309094 5612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/master-0/wal/instance:
uuid: "bba60aadb5c245e6bb6d4680cdfa0adc"
format_stamp: "Formatted at 2026-09-25 15:20:18 on dist-test-slave-4bk3"
server_key: "07fc9eef175733eeac3e24e01b383947"
server_key_iv: "1b8a2e8bbcbdcd4848d9e8f74e103a9b"
server_key_version: "encryptionkey@0"
I20260925 15:20:18.310520 5612 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:20:18.311326 5627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:18.311527 5612 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:18.311615 5612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/master-0/wal
uuid: "bba60aadb5c245e6bb6d4680cdfa0adc"
format_stamp: "Formatted at 2026-09-25 15:20:18 on dist-test-slave-4bk3"
server_key: "07fc9eef175733eeac3e24e01b383947"
server_key_iv: "1b8a2e8bbcbdcd4848d9e8f74e103a9b"
server_key_version: "encryptionkey@0"
I20260925 15:20:18.311684 5612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:18.346589 5612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:18.349030 5612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:18.349157 5612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:18.353552 5612 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.254:46877
I20260925 15:20:18.353642 5689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.254:46877 every 8 connection(s)
I20260925 15:20:18.354136 5612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
I20260925 15:20:18.354903 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:18.357398 5690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bba60aadb5c245e6bb6d4680cdfa0adc: Bootstrap starting.
I20260925 15:20:18.357944 5690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bba60aadb5c245e6bb6d4680cdfa0adc: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:18.358265 5690 log.cc:824] T 00000000000000000000000000000000 P bba60aadb5c245e6bb6d4680cdfa0adc: Log is configured to *not* fsync() on all Append() calls
I20260925 15:20:18.358989 5690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bba60aadb5c245e6bb6d4680cdfa0adc: No bootstrap required, opened a new log
I20260925 15:20:18.359892 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 5612
I20260925 15:20:18.359998 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/master-0/wal/instance
I20260925 15:20:18.360167 3919 external_mini_cluster.cc:1428] Setting key 2dd6b4c53d7d19c486140eca3112136d
I20260925 15:20:18.361821 5690 raft_consensus.cc:348] T 00000000000000000000000000000000 P bba60aadb5c245e6bb6d4680cdfa0adc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bba60aadb5c245e6bb6d4680cdfa0adc" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 46877 } }
I20260925 15:20:18.362017 5690 raft_consensus.cc:374] T 00000000000000000000000000000000 P bba60aadb5c245e6bb6d4680cdfa0adc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:18.362067 5690 raft_consensus.cc:729] T 00000000000000000000000000000000 P bba60aadb5c245e6bb6d4680cdfa0adc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bba60aadb5c245e6bb6d4680cdfa0adc, State: Initialized, Role: FOLLOWER
I20260925 15:20:18.362229 5690 consensus_queue.cc:260] T 00000000000000000000000000000000 P bba60aadb5c245e6bb6d4680cdfa0adc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bba60aadb5c245e6bb6d4680cdfa0adc" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 46877 } }
I20260925 15:20:18.362300 5690 raft_consensus.cc:388] T 00000000000000000000000000000000 P bba60aadb5c245e6bb6d4680cdfa0adc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:20:18.362323 5690 raft_consensus.cc:482] T 00000000000000000000000000000000 P bba60aadb5c245e6bb6d4680cdfa0adc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:20:18.362381 5690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bba60aadb5c245e6bb6d4680cdfa0adc [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:18.363348 5690 raft_consensus.cc:504] T 00000000000000000000000000000000 P bba60aadb5c245e6bb6d4680cdfa0adc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bba60aadb5c245e6bb6d4680cdfa0adc" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 46877 } }
I20260925 15:20:18.363469 5690 leader_election.cc:302] T 00000000000000000000000000000000 P bba60aadb5c245e6bb6d4680cdfa0adc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bba60aadb5c245e6bb6d4680cdfa0adc; no voters:
I20260925 15:20:18.363703 5690 leader_election.cc:288] T 00000000000000000000000000000000 P bba60aadb5c245e6bb6d4680cdfa0adc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:20:18.363778 5695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bba60aadb5c245e6bb6d4680cdfa0adc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:20:18.363968 5690 sys_catalog.cc:564] T 00000000000000000000000000000000 P bba60aadb5c245e6bb6d4680cdfa0adc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:20:18.363982 5695 raft_consensus.cc:686] T 00000000000000000000000000000000 P bba60aadb5c245e6bb6d4680cdfa0adc [term 1 LEADER]: Becoming Leader. State: Replica: bba60aadb5c245e6bb6d4680cdfa0adc, State: Running, Role: LEADER
I20260925 15:20:18.364738 5695 consensus_queue.cc:237] T 00000000000000000000000000000000 P bba60aadb5c245e6bb6d4680cdfa0adc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bba60aadb5c245e6bb6d4680cdfa0adc" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 46877 } }
I20260925 15:20:18.365371 5696 sys_catalog.cc:455] T 00000000000000000000000000000000 P bba60aadb5c245e6bb6d4680cdfa0adc [sys.catalog]: SysCatalogTable state changed. Reason: New leader bba60aadb5c245e6bb6d4680cdfa0adc. Latest consensus state: current_term: 1 leader_uuid: "bba60aadb5c245e6bb6d4680cdfa0adc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bba60aadb5c245e6bb6d4680cdfa0adc" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 46877 } } }
I20260925 15:20:18.365468 5696 sys_catalog.cc:458] T 00000000000000000000000000000000 P bba60aadb5c245e6bb6d4680cdfa0adc [sys.catalog]: This master's current role is: LEADER
W20260925 15:20:18.366189 5709 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bba60aadb5c245e6bb6d4680cdfa0adc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:20:18.366945 5709 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:20:18.366447 5697 sys_catalog.cc:455] T 00000000000000000000000000000000 P bba60aadb5c245e6bb6d4680cdfa0adc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bba60aadb5c245e6bb6d4680cdfa0adc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bba60aadb5c245e6bb6d4680cdfa0adc" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 46877 } } }
I20260925 15:20:18.367020 5710 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:20:18.367055 5697 sys_catalog.cc:458] T 00000000000000000000000000000000 P bba60aadb5c245e6bb6d4680cdfa0adc [sys.catalog]: This master's current role is: LEADER
I20260925 15:20:18.367206 5710 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 15:20:18 dist-test-slave-4bk3 krb5kdc[5599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:20:18.368754 5710 catalog_manager.cc:1347] Generated new cluster ID: 17d53daa5d474806bd272195c33a4d49
I20260925 15:20:18.368829 5710 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:20:18.373164 5694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:18.360734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58212 (local address 127.3.211.254:46877)
0925 15:20:18.360950 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:18.360963 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:18.361008 (+ 45us) server_negotiation.cc:407] Connection header received
0925 15:20:18.361770 (+ 762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:18.361780 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:18.361847 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:18.362027 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:18.362896 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:18.364431 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:18.365651 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:18.366120 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:18.366268 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:18.369064 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:18.369068 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:18.369069 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:18.369084 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:18.371181 (+ 2097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:18.371754 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:18.371759 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:18.371759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:18.371893 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:18.372125 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:18.372126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:18.372126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:18.372493 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:18.372659 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:18.372794 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 15:20:18.372939 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":92,"threads_started":1}
I20260925 15:20:18.385844 5710 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:20:18.386636 5710 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:20:18.396778 5710 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bba60aadb5c245e6bb6d4680cdfa0adc: Generated new TSK 0
I20260925 15:20:18.397054 5710 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.211.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab.
Entry for principal kudu/127.3.211.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab.
I20260925 15:20:18.423517 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.193:0
--local_ip_for_outbound_sockets=127.3.211.193
--webserver_interface=127.3.211.193
--webserver_port=0
--tserver_master_addrs=127.3.211.254:46877
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab
--principal=kudu/127.3.211.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:34273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:20:18.512633 5716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:18.512881 5716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:18.512914 5716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:18.514346 5716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:18.514410 5716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:18.514468 5716 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 15:20:18.514489 5716 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:20:18.514529 5716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.193
I20260925 15:20:18.516556 5716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:34273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab
--principal=kudu/127.3.211.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.211.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.211.254:46877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.193
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:18.516866 5716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:18.517167 5716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:18.522869 5721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:18.522869 5724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:18.523104 5722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:18.523856 5716 server_base.cc:1034] running on GCE node
I20260925 15:20:18.524124 5716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:18.524475 5716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:18.525662 5716 hybrid_clock.cc:648] HybridClock initialized: now 1790349618525595 us; error 85 us; skew 500 ppm
Sep 25 15:20:18 dist-test-slave-4bk3 krb5kdc[5599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349618, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:18.528491 5716 init.cc:377] Logged in from keytab as kudu/127.3.211.193@KRBTEST.COM (short username kudu)
I20260925 15:20:18.529038 5716 webserver.cc:460] Webserver started at http://127.3.211.193:43397/ using document root <none> and password file <none>
I20260925 15:20:18.529316 5716 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:18.529392 5716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:18.529552 5716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:18.530696 5716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-0/data/instance:
uuid: "9ef0e719f724454ba531038d0612a66b"
format_stamp: "Formatted at 2026-09-25 15:20:18 on dist-test-slave-4bk3"
server_key: "b96ec19439ee8e991377acbab7b3ff87"
server_key_iv: "272748242c7abf7cbfa931510c3a76fe"
server_key_version: "encryptionkey@0"
I20260925 15:20:18.531081 5716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-0/wal/instance:
uuid: "9ef0e719f724454ba531038d0612a66b"
format_stamp: "Formatted at 2026-09-25 15:20:18 on dist-test-slave-4bk3"
server_key: "b96ec19439ee8e991377acbab7b3ff87"
server_key_iv: "272748242c7abf7cbfa931510c3a76fe"
server_key_version: "encryptionkey@0"
I20260925 15:20:18.532661 5716 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:20:18.533488 5731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:18.533694 5716 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:18.533795 5716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-0/wal
uuid: "9ef0e719f724454ba531038d0612a66b"
format_stamp: "Formatted at 2026-09-25 15:20:18 on dist-test-slave-4bk3"
server_key: "b96ec19439ee8e991377acbab7b3ff87"
server_key_iv: "272748242c7abf7cbfa931510c3a76fe"
server_key_version: "encryptionkey@0"
I20260925 15:20:18.533883 5716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:18.543577 5716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:18.546197 5716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:18.546380 5716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:18.547019 5716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:20:18.547071 5716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:18.547103 5716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:20:18.547124 5716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 15:20:18 dist-test-slave-4bk3 krb5kdc[5599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349618, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.193@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:20:18.554590 5716 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.193:34301
I20260925 15:20:18.554632 5846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.193:34301 every 8 connection(s)
I20260925 15:20:18.555217 5716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:20:18.558225 5694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:18.547575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:43259 (local address 127.3.211.254:46877)
0925 15:20:18.547641 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:18.547643 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:18.547876 (+ 233us) server_negotiation.cc:407] Connection header received
0925 15:20:18.548381 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:18.548383 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:18.548401 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:18.548462 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:18.549228 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:18.549733 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:18.551416 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:18.551807 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:18.551900 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:18.554933 (+ 3033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:18.554935 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:18.554935 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:18.554947 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:18.556704 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:18.557185 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:18.557187 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:18.557187 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:18.557260 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:18.557558 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:18.557559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:18.557559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:18.557697 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:18.557752 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:18.558068 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 15:20:18.558157 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 15:20:18.558357 5740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:18.547563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:46877 (local address 127.3.211.193:43259)
0925 15:20:18.547729 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:18.547770 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:18.548259 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:18.548543 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:18.548546 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:18.548912 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:18.549119 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:18.549124 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:18.549834 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:18.549835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:18.551322 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:18.551325 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:18.551924 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:18.551925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:18.552061 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:18.552676 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:18.552688 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:18.554730 (+ 2042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:18.556820 (+ 2090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:18.556827 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:18.556829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:18.557076 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:18.557331 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:18.557332 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:18.557332 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:18.557450 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:18.557764 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:18.557770 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:18.557901 (+ 131us) client_negotiation.cc:769] Sending connection context
0925 15:20:18.558012 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 15:20:18.558160 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":94,"threads_started":1}
I20260925 15:20:18.558907 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 5716
I20260925 15:20:18.558912 5847 heartbeater.cc:347] Connected to a master server at 127.3.211.254:46877
I20260925 15:20:18.558990 5847 heartbeater.cc:464] Registering TS with master...
I20260925 15:20:18.559007 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-0/wal/instance
I20260925 15:20:18.559187 3919 external_mini_cluster.cc:1428] Setting key 9344ebbe13c4a4b3395d86909d99d5ad
I20260925 15:20:18.559185 5847 heartbeater.cc:511] Master 127.3.211.254:46877 requested a full tablet report, sending...
I20260925 15:20:18.559722 5643 ts_manager.cc:194] Registered new tserver with Master: 9ef0e719f724454ba531038d0612a66b (127.3.211.193:34301)
I20260925 15:20:18.560593 5643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.193@KRBTEST.COM'} at 127.3.211.193:43259
I20260925 15:20:18.567179 5694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:18.560045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:34633 (local address 127.3.211.254:46877)
0925 15:20:18.560085 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:18.560087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:18.560148 (+ 61us) server_negotiation.cc:407] Connection header received
0925 15:20:18.560243 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:18.560244 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:18.560253 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:18.560307 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:18.560740 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:18.561460 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:18.562414 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:18.562732 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:18.562816 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:18.564088 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:18.564089 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:18.564090 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:18.564097 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:18.565755 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:18.566195 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:18.566196 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:18.566197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:18.566271 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:18.566587 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:18.566588 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:18.566588 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:18.566726 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:18.566790 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:18.567023 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 15:20:18.567104 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 15:20:18.567201 5853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:18.559985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:46877 (local address 127.3.211.193:34633)
0925 15:20:18.560096 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:18.560106 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:18.560166 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:18.560398 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:18.560399 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:18.560593 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:18.560644 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:18.560646 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:18.561565 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:18.561566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:18.562320 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:18.562322 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:18.562815 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:18.562816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:18.562871 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:18.563391 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:18.563396 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:18.563826 (+ 430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:18.565838 (+ 2012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:18.565839 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:18.565840 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:18.566094 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:18.566352 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:18.566353 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:18.566354 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:18.566434 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:18.566803 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:18.566804 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:18.566864 (+ 60us) client_negotiation.cc:769] Sending connection context
0925 15:20:18.566947 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 15:20:18.567071 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":63,"threads_started":1}
WARNING: no policy specified for kudu/127.3.211.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab.
Entry for principal kudu/127.3.211.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab.
I20260925 15:20:18.588050 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.194:0
--local_ip_for_outbound_sockets=127.3.211.194
--webserver_interface=127.3.211.194
--webserver_port=0
--tserver_master_addrs=127.3.211.254:46877
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab
--principal=kudu/127.3.211.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:34273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:20:18.670506 5856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:18.670783 5856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:18.670812 5856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:18.672060 5856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:18.672111 5856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:18.672132 5856 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 15:20:18.672144 5856 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:20:18.672163 5856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.194
I20260925 15:20:18.673684 5856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:34273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab
--principal=kudu/127.3.211.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.211.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.211.254:46877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.194
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:18.673897 5856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:18.674086 5856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:18.679324 5861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:18.679373 5862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:18.679571 5864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:18.679804 5856 server_base.cc:1034] running on GCE node
I20260925 15:20:18.680025 5856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:18.680307 5856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:18.681523 5856 hybrid_clock.cc:648] HybridClock initialized: now 1790349618681502 us; error 37 us; skew 500 ppm
Sep 25 15:20:18 dist-test-slave-4bk3 krb5kdc[5599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349618, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:18.684094 5856 init.cc:377] Logged in from keytab as kudu/127.3.211.194@KRBTEST.COM (short username kudu)
I20260925 15:20:18.684645 5856 webserver.cc:460] Webserver started at http://127.3.211.194:33107/ using document root <none> and password file <none>
I20260925 15:20:18.684958 5856 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:18.685067 5856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:18.685238 5856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:18.686311 5856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-1/data/instance:
uuid: "ddb9ca84c5f740ba8983704a4381c165"
format_stamp: "Formatted at 2026-09-25 15:20:18 on dist-test-slave-4bk3"
server_key: "f54a6938f5856e2492cd287186d05c4c"
server_key_iv: "66bbe9da5738d6ecf18ad28f4f7e24ca"
server_key_version: "encryptionkey@0"
I20260925 15:20:18.686673 5856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-1/wal/instance:
uuid: "ddb9ca84c5f740ba8983704a4381c165"
format_stamp: "Formatted at 2026-09-25 15:20:18 on dist-test-slave-4bk3"
server_key: "f54a6938f5856e2492cd287186d05c4c"
server_key_iv: "66bbe9da5738d6ecf18ad28f4f7e24ca"
server_key_version: "encryptionkey@0"
I20260925 15:20:18.688046 5856 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:20:18.688936 5871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:18.689146 5856 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 15:20:18.689252 5856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-1/wal
uuid: "ddb9ca84c5f740ba8983704a4381c165"
format_stamp: "Formatted at 2026-09-25 15:20:18 on dist-test-slave-4bk3"
server_key: "f54a6938f5856e2492cd287186d05c4c"
server_key_iv: "66bbe9da5738d6ecf18ad28f4f7e24ca"
server_key_version: "encryptionkey@0"
I20260925 15:20:18.689337 5856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:18.712970 5856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:18.715593 5856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:18.715778 5856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:18.716377 5856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:20:18.716462 5856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:18.716498 5856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:20:18.716537 5856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:18.723639 5856 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.194:35019
Sep 25 15:20:18 dist-test-slave-4bk3 krb5kdc[5599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349618, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.194@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:20:18.723711 5986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.194:35019 every 8 connection(s)
I20260925 15:20:18.724072 5856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
I20260925 15:20:18.726749 5694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:18.716936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:36805 (local address 127.3.211.254:46877)
0925 15:20:18.717006 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:18.717009 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:18.717253 (+ 244us) server_negotiation.cc:407] Connection header received
0925 15:20:18.717754 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:18.717755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:18.717770 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:18.717864 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:18.718818 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:18.719653 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:18.720994 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:18.721212 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:18.721311 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:18.724194 (+ 2883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:18.724196 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:18.724197 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:18.724206 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:18.725560 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:18.725976 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:18.725976 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:18.725977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:18.726036 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:18.726243 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:18.726244 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:18.726244 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:18.726354 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:18.726400 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:18.726619 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 15:20:18.726683 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 15:20:18.727023 5880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:18.716935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:46877 (local address 127.3.211.194:36805)
0925 15:20:18.717118 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:18.717160 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:18.717658 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:18.717937 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:18.717941 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:18.718361 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:18.718717 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:18.718725 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:18.719763 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:18.719764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:18.720882 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:18.720885 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:18.721343 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:18.721344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:18.721594 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:18.722489 (+ 895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:18.722498 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:18.724075 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:18.725668 (+ 1593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:18.725674 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:18.725676 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:18.725903 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:18.726095 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:18.726096 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:18.726096 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:18.726190 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:18.726398 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:18.726400 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:18.726516 (+ 116us) client_negotiation.cc:769] Sending connection context
0925 15:20:18.726595 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 15:20:18.726742 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":93,"threads_started":1}
I20260925 15:20:18.727660 5987 heartbeater.cc:347] Connected to a master server at 127.3.211.254:46877
I20260925 15:20:18.727763 5987 heartbeater.cc:464] Registering TS with master...
I20260925 15:20:18.727942 5987 heartbeater.cc:511] Master 127.3.211.254:46877 requested a full tablet report, sending...
I20260925 15:20:18.728390 5644 ts_manager.cc:194] Registered new tserver with Master: ddb9ca84c5f740ba8983704a4381c165 (127.3.211.194:35019)
I20260925 15:20:18.728912 5644 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.194@KRBTEST.COM'} at 127.3.211.194:36805
I20260925 15:20:18.733691 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 5856
I20260925 15:20:18.733793 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-1/wal/instance
I20260925 15:20:18.733989 3919 external_mini_cluster.cc:1428] Setting key df604312dfaf440eb8e7025bacfa7666
I20260925 15:20:18.735213 5694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:18.728805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:39289 (local address 127.3.211.254:46877)
0925 15:20:18.729048 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:18.729050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:18.729058 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:20:18.729076 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:18.729077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:18.729087 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:18.729144 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:18.729501 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:18.729975 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:18.730714 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:18.730917 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:18.730991 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:18.732023 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:18.732025 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:18.732026 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:18.732032 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:18.733821 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:18.734253 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:18.734254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:18.734255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:18.734328 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:18.734564 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:18.734565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:18.734566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:18.734773 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:18.734859 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:18.735016 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 15:20:18.735112 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224}
I20260925 15:20:18.735221 5993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:18.728715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:46877 (local address 127.3.211.194:39289)
0925 15:20:18.728876 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:18.728887 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:18.728946 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:18.729160 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:18.729161 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:18.729349 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:18.729418 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:18.729419 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:18.730076 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:18.730077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:18.730601 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:18.730603 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:18.731000 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:18.731001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:18.731074 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:18.731564 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:18.731569 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:18.731899 (+ 330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:18.733936 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:18.733937 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:18.733937 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:18.734165 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:18.734420 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:18.734421 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:18.734422 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:18.734494 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:18.734848 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:18.734849 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:18.734898 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 15:20:18.734990 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 15:20:18.735113 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":100,"threads_started":1}
WARNING: no policy specified for kudu/127.3.211.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab.
Entry for principal kudu/127.3.211.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab.
I20260925 15:20:18.759749 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.195:0
--local_ip_for_outbound_sockets=127.3.211.195
--webserver_interface=127.3.211.195
--webserver_port=0
--tserver_master_addrs=127.3.211.254:46877
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab
--principal=kudu/127.3.211.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:34273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:20:18.844349 5996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:18.844559 5996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:18.844586 5996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:18.845817 5996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:18.845867 5996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:18.845890 5996 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 15:20:18.845902 5996 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:20:18.845950 5996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.195
I20260925 15:20:18.847458 5996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:34273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab
--principal=kudu/127.3.211.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.211.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.211.254:46877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.195
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:18.847710 5996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:18.847982 5996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:18.853781 6001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:18.853829 6004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:18.853781 6002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:18.854147 5996 server_base.cc:1034] running on GCE node
I20260925 15:20:18.854461 5996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:18.854754 5996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:18.855971 5996 hybrid_clock.cc:648] HybridClock initialized: now 1790349618855959 us; error 36 us; skew 500 ppm
Sep 25 15:20:18 dist-test-slave-4bk3 krb5kdc[5599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349618, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:18.858906 5996 init.cc:377] Logged in from keytab as kudu/127.3.211.195@KRBTEST.COM (short username kudu)
I20260925 15:20:18.859486 5996 webserver.cc:460] Webserver started at http://127.3.211.195:37085/ using document root <none> and password file <none>
I20260925 15:20:18.859768 5996 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:18.859843 5996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:18.859983 5996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:18.861159 5996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-2/data/instance:
uuid: "074cc39baf68454099b816ad22fddc74"
format_stamp: "Formatted at 2026-09-25 15:20:18 on dist-test-slave-4bk3"
server_key: "5e481eee1c3e817d9ba8cb448f1a4413"
server_key_iv: "155a7d2a44c3462bd471f096bfb46467"
server_key_version: "encryptionkey@0"
I20260925 15:20:18.861557 5996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-2/wal/instance:
uuid: "074cc39baf68454099b816ad22fddc74"
format_stamp: "Formatted at 2026-09-25 15:20:18 on dist-test-slave-4bk3"
server_key: "5e481eee1c3e817d9ba8cb448f1a4413"
server_key_iv: "155a7d2a44c3462bd471f096bfb46467"
server_key_version: "encryptionkey@0"
I20260925 15:20:18.862933 5996 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 15:20:18.863763 6011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:18.863965 5996 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:20:18.864068 5996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-2/wal
uuid: "074cc39baf68454099b816ad22fddc74"
format_stamp: "Formatted at 2026-09-25 15:20:18 on dist-test-slave-4bk3"
server_key: "5e481eee1c3e817d9ba8cb448f1a4413"
server_key_iv: "155a7d2a44c3462bd471f096bfb46467"
server_key_version: "encryptionkey@0"
I20260925 15:20:18.864142 5996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:18.904587 5996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:18.906713 5996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:18.906828 5996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:18.907303 5996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:20:18.907380 5996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:18.907428 5996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:20:18.907456 5996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:18.914466 5996 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.195:38695
I20260925 15:20:18.914541 6126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.195:38695 every 8 connection(s)
Sep 25 15:20:18 dist-test-slave-4bk3 krb5kdc[5599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349618, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.195@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:20:18.914906 5996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
I20260925 15:20:18.915870 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 5996
I20260925 15:20:18.915961 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-2/wal/instance
I20260925 15:20:18.916148 3919 external_mini_cluster.cc:1428] Setting key 746234c43614ab57b182e16ea5306e39
I20260925 15:20:18.918962 6020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:18.907842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:46877 (local address 127.3.211.195:45969)
0925 15:20:18.907998 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:18.908039 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:18.908624 (+ 585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:18.908903 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:18.908907 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:18.909292 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:18.909605 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:18.909610 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:18.910683 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:18.910685 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:18.912080 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:18.912084 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:18.912635 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:18.912636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:18.912740 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:18.913346 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:18.913359 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:18.915476 (+ 2117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:18.917439 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:18.917446 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:18.917448 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:18.917687 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:18.917910 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:18.917911 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:18.917912 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:18.918031 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:18.918377 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:18.918383 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:18.918514 (+ 131us) client_negotiation.cc:769] Sending connection context
0925 15:20:18.918625 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 15:20:18.918766 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":89,"threads_started":1}
I20260925 15:20:18.919127 5694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:18.907878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:45969 (local address 127.3.211.254:46877)
0925 15:20:18.907950 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:18.907953 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:18.908135 (+ 182us) server_negotiation.cc:407] Connection header received
0925 15:20:18.908726 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:18.908727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:18.908740 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:18.908825 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:18.909730 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:18.910573 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:18.912201 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:18.912542 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:18.912633 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:18.915594 (+ 2961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:18.915596 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:18.915596 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:18.915605 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:18.917355 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:18.917762 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:18.917763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:18.917763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:18.917819 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:18.918127 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:18.918128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:18.918128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:18.918277 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:18.918344 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:18.918776 (+ 432us) server_negotiation.cc:299] Negotiation successful
0925 15:20:18.918909 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 15:20:18.919837 6127 heartbeater.cc:347] Connected to a master server at 127.3.211.254:46877
I20260925 15:20:18.919945 6127 heartbeater.cc:464] Registering TS with master...
I20260925 15:20:18.920153 6127 heartbeater.cc:511] Master 127.3.211.254:46877 requested a full tablet report, sending...
I20260925 15:20:18.920656 5643 ts_manager.cc:194] Registered new tserver with Master: 074cc39baf68454099b816ad22fddc74 (127.3.211.195:38695)
I20260925 15:20:18.921088 5643 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.195@KRBTEST.COM'} at 127.3.211.195:45969
I20260925 15:20:18.926412 6133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:18.921021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:46877 (local address 127.3.211.195:58257)
0925 15:20:18.921200 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:18.921211 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:18.921267 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:18.921406 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:18.921406 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:18.921654 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:18.921716 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:18.921718 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:18.922353 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:18.922354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:18.922847 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:18.922852 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:18.923235 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:18.923236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:18.923300 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:18.923743 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:18.923747 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:18.924038 (+ 291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:18.925504 (+ 1466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:18.925505 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:18.925505 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:18.925672 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:18.925839 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:18.925840 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:18.925840 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:18.925890 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:18.926115 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:18.926115 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:18.926163 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 15:20:18.926236 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 15:20:18.926331 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":133,"threads_started":1}
I20260925 15:20:18.926433 5694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:18.921069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:58257 (local address 127.3.211.254:46877)
0925 15:20:18.921115 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:18.921118 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:18.921274 (+ 156us) server_negotiation.cc:407] Connection header received
0925 15:20:18.921326 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:18.921327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:18.921338 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:18.921392 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:18.921796 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:18.922263 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:18.922952 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:18.923147 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:18.923208 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:18.924128 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:18.924130 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:18.924130 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:18.924136 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:18.925428 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:18.925736 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:18.925736 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:18.925737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:18.925783 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:18.925939 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:18.925939 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:18.925940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:18.926034 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:18.926083 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:18.926293 (+ 210us) server_negotiation.cc:299] Negotiation successful
0925 15:20:18.926363 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 15:20:18.927940 3919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 15:20:18 dist-test-slave-4bk3 krb5kdc[5599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 15:20:18 dist-test-slave-4bk3 krb5kdc[5599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:20:18.952013 5694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:18.944476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58216 (local address 127.3.211.254:46877)
0925 15:20:18.944542 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:18.944544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:18.944572 (+ 28us) server_negotiation.cc:407] Connection header received
0925 15:20:18.944699 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:18.944700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:18.944711 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:18.944780 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:18.945377 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:18.945880 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:18.946632 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:18.946828 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:18.946928 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:18.949446 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:18.949448 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:18.949448 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:18.949458 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:18.950696 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:18.951167 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:18.951168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:18.951168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:18.951250 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:18.951461 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:18.951462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:18.951462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:18.951584 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:18.951691 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:18.951819 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 15:20:18.951905 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 15:20:18.954023 5643 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58216:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:20:18.955649 5643 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:20:18.962980 6147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:18.958538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59108 (local address 127.3.211.195:38695)
0925 15:20:18.958740 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:18.958746 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:18.958757 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:20:18.958834 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:18.958836 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:18.958850 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:18.958946 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:20:18.959268 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:18.960404 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:18.961984 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:18.962501 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:18.962599 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:18.962658 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:18.962854 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 15:20:18.962894 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":83,"threads_started":1}
I20260925 15:20:18.962970 6146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:18.958292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:34301 (local address 127.0.0.1:52376)
0925 15:20:18.958711 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:18.958721 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:18.958778 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:18.959017 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:18.959018 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:18.959020 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:20:18.959078 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:18.959080 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:18.960481 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:18.960483 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:18.961641 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:18.961643 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:18.962588 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:18.962589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:18.962698 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:18.962715 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 15:20:18.962792 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 15:20:18.962872 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"mutex_wait_us":34,"thread_start_us":358,"threads_started":1}
I20260925 15:20:18.963405 6144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:18.958211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.194:35019 (local address 127.0.0.1:47456)
0925 15:20:18.958384 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:18.958473 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:18.958543 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:18.959069 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:18.959070 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:18.959072 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:20:18.959102 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:18.959103 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:18.960600 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:18.960601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:18.961879 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:18.961880 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:18.962775 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:18.962776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:18.962842 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:18.962847 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:20:18.962905 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 15:20:18.962951 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":108,"thread_start_us":62,"threads_started":1}
I20260925 15:20:18.963419 6145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:18.958300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47456 (local address 127.3.211.194:35019)
0925 15:20:18.958890 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:18.958894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:18.958905 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:20:18.958937 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:18.958939 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:18.958953 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:18.959195 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:20:18.959314 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:18.960440 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:18.961969 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:18.962681 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:18.962774 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:18.962825 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:18.962919 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 15:20:18.962958 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":84,"threads_started":1}
I20260925 15:20:18.963397 6149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:18.958614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52376 (local address 127.3.211.193:34301)
0925 15:20:18.958808 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:18.958812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:18.958823 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:20:18.958862 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:18.958864 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:18.958879 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:18.958955 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:20:18.959146 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:18.960406 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:18.961735 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:18.962479 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:18.962576 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:18.962638 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:18.963280 (+ 642us) server_negotiation.cc:299] Negotiation successful
0925 15:20:18.963326 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":89,"threads_started":1}
I20260925 15:20:18.963534 6148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:18.958289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.195:38695 (local address 127.0.0.1:59108)
0925 15:20:18.958681 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:18.958692 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:18.958769 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:18.958954 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:18.958957 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:18.958960 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:20:18.959063 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:18.959065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:18.960507 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:18.960508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:18.961877 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:18.961880 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:18.962692 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:18.962693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:18.962750 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:18.962754 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:20:18.962800 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 15:20:18.963143 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"mutex_wait_us":364,"thread_start_us":108,"threads_started":1}
I20260925 15:20:18.964370 5921 tablet_service.cc:1467] Processing CreateTablet for tablet b4f369beb03642a5bc095b07cb6b2ece (DEFAULT_TABLE table=test-table [id=77726c28fc6a4a83afc4961448849e97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:20:18.964635 5781 tablet_service.cc:1467] Processing CreateTablet for tablet b4f369beb03642a5bc095b07cb6b2ece (DEFAULT_TABLE table=test-table [id=77726c28fc6a4a83afc4961448849e97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:20:18.964634 6061 tablet_service.cc:1467] Processing CreateTablet for tablet b4f369beb03642a5bc095b07cb6b2ece (DEFAULT_TABLE table=test-table [id=77726c28fc6a4a83afc4961448849e97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:20:18.964816 5921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4f369beb03642a5bc095b07cb6b2ece. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:18.964941 5781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4f369beb03642a5bc095b07cb6b2ece. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:18.965042 6061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4f369beb03642a5bc095b07cb6b2ece. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:18.967659 6150 tablet_bootstrap.cc:492] T b4f369beb03642a5bc095b07cb6b2ece P 074cc39baf68454099b816ad22fddc74: Bootstrap starting.
I20260925 15:20:18.968781 6151 tablet_bootstrap.cc:492] T b4f369beb03642a5bc095b07cb6b2ece P ddb9ca84c5f740ba8983704a4381c165: Bootstrap starting.
I20260925 15:20:18.968854 6150 tablet_bootstrap.cc:654] T b4f369beb03642a5bc095b07cb6b2ece P 074cc39baf68454099b816ad22fddc74: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:18.969048 6152 tablet_bootstrap.cc:492] T b4f369beb03642a5bc095b07cb6b2ece P 9ef0e719f724454ba531038d0612a66b: Bootstrap starting.
I20260925 15:20:18.969281 6150 log.cc:824] T b4f369beb03642a5bc095b07cb6b2ece P 074cc39baf68454099b816ad22fddc74: Log is configured to *not* fsync() on all Append() calls
I20260925 15:20:18.969507 6151 tablet_bootstrap.cc:654] T b4f369beb03642a5bc095b07cb6b2ece P ddb9ca84c5f740ba8983704a4381c165: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:18.969702 6152 tablet_bootstrap.cc:654] T b4f369beb03642a5bc095b07cb6b2ece P 9ef0e719f724454ba531038d0612a66b: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:18.969836 6151 log.cc:824] T b4f369beb03642a5bc095b07cb6b2ece P ddb9ca84c5f740ba8983704a4381c165: Log is configured to *not* fsync() on all Append() calls
I20260925 15:20:18.970026 6152 log.cc:824] T b4f369beb03642a5bc095b07cb6b2ece P 9ef0e719f724454ba531038d0612a66b: Log is configured to *not* fsync() on all Append() calls
I20260925 15:20:18.970160 6150 tablet_bootstrap.cc:492] T b4f369beb03642a5bc095b07cb6b2ece P 074cc39baf68454099b816ad22fddc74: No bootstrap required, opened a new log
I20260925 15:20:18.970268 6150 ts_tablet_manager.cc:1397] T b4f369beb03642a5bc095b07cb6b2ece P 074cc39baf68454099b816ad22fddc74: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260925 15:20:18.970683 6151 tablet_bootstrap.cc:492] T b4f369beb03642a5bc095b07cb6b2ece P ddb9ca84c5f740ba8983704a4381c165: No bootstrap required, opened a new log
I20260925 15:20:18.970753 6151 ts_tablet_manager.cc:1397] T b4f369beb03642a5bc095b07cb6b2ece P ddb9ca84c5f740ba8983704a4381c165: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:20:18.972374 6150 raft_consensus.cc:348] T b4f369beb03642a5bc095b07cb6b2ece P 074cc39baf68454099b816ad22fddc74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } } peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } }
I20260925 15:20:18.972568 6150 raft_consensus.cc:374] T b4f369beb03642a5bc095b07cb6b2ece P 074cc39baf68454099b816ad22fddc74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:18.972604 6150 raft_consensus.cc:729] T b4f369beb03642a5bc095b07cb6b2ece P 074cc39baf68454099b816ad22fddc74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074cc39baf68454099b816ad22fddc74, State: Initialized, Role: FOLLOWER
I20260925 15:20:18.972728 6150 consensus_queue.cc:260] T b4f369beb03642a5bc095b07cb6b2ece P 074cc39baf68454099b816ad22fddc74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } } peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } }
I20260925 15:20:18.972770 6151 raft_consensus.cc:348] T b4f369beb03642a5bc095b07cb6b2ece P ddb9ca84c5f740ba8983704a4381c165 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } } peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } }
I20260925 15:20:18.972932 6151 raft_consensus.cc:374] T b4f369beb03642a5bc095b07cb6b2ece P ddb9ca84c5f740ba8983704a4381c165 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:18.972975 6151 raft_consensus.cc:729] T b4f369beb03642a5bc095b07cb6b2ece P ddb9ca84c5f740ba8983704a4381c165 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddb9ca84c5f740ba8983704a4381c165, State: Initialized, Role: FOLLOWER
I20260925 15:20:18.973016 6150 ts_tablet_manager.cc:1428] T b4f369beb03642a5bc095b07cb6b2ece P 074cc39baf68454099b816ad22fddc74: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 15:20:18.973146 6127 heartbeater.cc:503] Master 127.3.211.254:46877 was elected leader, sending a full tablet report...
I20260925 15:20:18.973121 6151 consensus_queue.cc:260] T b4f369beb03642a5bc095b07cb6b2ece P ddb9ca84c5f740ba8983704a4381c165 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } } peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } }
I20260925 15:20:18.973179 6152 tablet_bootstrap.cc:492] T b4f369beb03642a5bc095b07cb6b2ece P 9ef0e719f724454ba531038d0612a66b: No bootstrap required, opened a new log
I20260925 15:20:18.973304 6152 ts_tablet_manager.cc:1397] T b4f369beb03642a5bc095b07cb6b2ece P 9ef0e719f724454ba531038d0612a66b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 15:20:18.973392 6151 ts_tablet_manager.cc:1428] T b4f369beb03642a5bc095b07cb6b2ece P ddb9ca84c5f740ba8983704a4381c165: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:20:18.973441 5987 heartbeater.cc:503] Master 127.3.211.254:46877 was elected leader, sending a full tablet report...
W20260925 15:20:18.974911 5988 tablet.cc:2369] T b4f369beb03642a5bc095b07cb6b2ece P ddb9ca84c5f740ba8983704a4381c165: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:20:18.975376 6152 raft_consensus.cc:348] T b4f369beb03642a5bc095b07cb6b2ece P 9ef0e719f724454ba531038d0612a66b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } } peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } }
I20260925 15:20:18.975500 6152 raft_consensus.cc:374] T b4f369beb03642a5bc095b07cb6b2ece P 9ef0e719f724454ba531038d0612a66b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:18.975520 6152 raft_consensus.cc:729] T b4f369beb03642a5bc095b07cb6b2ece P 9ef0e719f724454ba531038d0612a66b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef0e719f724454ba531038d0612a66b, State: Initialized, Role: FOLLOWER
I20260925 15:20:18.975639 6152 consensus_queue.cc:260] T b4f369beb03642a5bc095b07cb6b2ece P 9ef0e719f724454ba531038d0612a66b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } } peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } }
I20260925 15:20:18.975869 6152 ts_tablet_manager.cc:1428] T b4f369beb03642a5bc095b07cb6b2ece P 9ef0e719f724454ba531038d0612a66b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 15:20:18.975965 5847 heartbeater.cc:503] Master 127.3.211.254:46877 was elected leader, sending a full tablet report...
W20260925 15:20:19.055994 5848 tablet.cc:2369] T b4f369beb03642a5bc095b07cb6b2ece P 9ef0e719f724454ba531038d0612a66b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:20:19.165577 6128 tablet.cc:2369] T b4f369beb03642a5bc095b07cb6b2ece P 074cc39baf68454099b816ad22fddc74: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:20:19.219929 6156 raft_consensus.cc:482] T b4f369beb03642a5bc095b07cb6b2ece P 074cc39baf68454099b816ad22fddc74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:20:19.220075 6156 raft_consensus.cc:504] T b4f369beb03642a5bc095b07cb6b2ece P 074cc39baf68454099b816ad22fddc74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } } peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } }
I20260925 15:20:19.220584 6156 leader_election.cc:288] T b4f369beb03642a5bc095b07cb6b2ece P 074cc39baf68454099b816ad22fddc74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddb9ca84c5f740ba8983704a4381c165 (127.3.211.194:35019), 9ef0e719f724454ba531038d0612a66b (127.3.211.193:34301)
I20260925 15:20:19.225116 6145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:19.220890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:44141 (local address 127.3.211.194:35019)
0925 15:20:19.220936 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:19.220938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:19.220951 (+ 13us) server_negotiation.cc:407] Connection header received
0925 15:20:19.221033 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:19.221034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:19.221041 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:19.221128 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:20:19.221313 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:19.222089 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:19.223965 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:19.224590 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:19.224669 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:19.224678 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:19.224990 (+ 312us) server_negotiation.cc:299] Negotiation successful
0925 15:20:19.225042 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 15:20:19.225121 6149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:19.220903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:48115 (local address 127.3.211.193:34301)
0925 15:20:19.220966 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:19.220969 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:19.221117 (+ 148us) server_negotiation.cc:407] Connection header received
0925 15:20:19.221168 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:19.221169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:19.221177 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:19.221231 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:20:19.221393 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:19.222089 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:19.223965 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:19.224577 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:19.224669 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:19.224678 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:19.224989 (+ 311us) server_negotiation.cc:299] Negotiation successful
0925 15:20:19.225040 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 15:20:19.225158 6020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:19.220817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.194:35019 (local address 127.3.211.195:44141)
0925 15:20:19.220878 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:19.220892 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:19.220941 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:19.221132 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:19.221133 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:19.221135 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:20:19.221231 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:19.221234 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:19.222222 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:19.222224 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:19.223768 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:19.223771 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:19.224692 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:19.224694 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:19.224799 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:19.224802 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:20:19.224884 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 15:20:19.224935 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":115}
I20260925 15:20:19.225175 6159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:19.220831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:34301 (local address 127.3.211.195:48115)
0925 15:20:19.221026 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:19.221035 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:19.221118 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:19.221254 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:19.221255 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:19.221256 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:20:19.221321 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:19.221322 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:19.222223 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:19.222224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:19.223768 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:19.223771 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:19.224692 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:19.224694 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:19.224786 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:19.224792 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 15:20:19.224879 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 15:20:19.224936 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":64,"thread_start_us":112,"threads_started":1}
I20260925 15:20:19.225656 5801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f369beb03642a5bc095b07cb6b2ece" candidate_uuid: "074cc39baf68454099b816ad22fddc74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef0e719f724454ba531038d0612a66b" is_pre_election: true
I20260925 15:20:19.225661 5941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f369beb03642a5bc095b07cb6b2ece" candidate_uuid: "074cc39baf68454099b816ad22fddc74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddb9ca84c5f740ba8983704a4381c165" is_pre_election: true
I20260925 15:20:19.225834 5801 raft_consensus.cc:2454] T b4f369beb03642a5bc095b07cb6b2ece P 9ef0e719f724454ba531038d0612a66b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074cc39baf68454099b816ad22fddc74 in term 0.
I20260925 15:20:19.225847 5941 raft_consensus.cc:2454] T b4f369beb03642a5bc095b07cb6b2ece P ddb9ca84c5f740ba8983704a4381c165 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074cc39baf68454099b816ad22fddc74 in term 0.
I20260925 15:20:19.226111 6012 leader_election.cc:302] T b4f369beb03642a5bc095b07cb6b2ece P 074cc39baf68454099b816ad22fddc74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074cc39baf68454099b816ad22fddc74, 9ef0e719f724454ba531038d0612a66b; no voters:
I20260925 15:20:19.226276 6156 raft_consensus.cc:2781] T b4f369beb03642a5bc095b07cb6b2ece P 074cc39baf68454099b816ad22fddc74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:20:19.226341 6156 raft_consensus.cc:482] T b4f369beb03642a5bc095b07cb6b2ece P 074cc39baf68454099b816ad22fddc74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:20:19.226359 6156 raft_consensus.cc:3037] T b4f369beb03642a5bc095b07cb6b2ece P 074cc39baf68454099b816ad22fddc74 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:19.227280 6156 raft_consensus.cc:504] T b4f369beb03642a5bc095b07cb6b2ece P 074cc39baf68454099b816ad22fddc74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } } peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } }
I20260925 15:20:19.227456 6156 leader_election.cc:288] T b4f369beb03642a5bc095b07cb6b2ece P 074cc39baf68454099b816ad22fddc74 [CANDIDATE]: Term 1 election: Requested vote from peers ddb9ca84c5f740ba8983704a4381c165 (127.3.211.194:35019), 9ef0e719f724454ba531038d0612a66b (127.3.211.193:34301)
I20260925 15:20:19.227640 5941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f369beb03642a5bc095b07cb6b2ece" candidate_uuid: "074cc39baf68454099b816ad22fddc74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddb9ca84c5f740ba8983704a4381c165"
I20260925 15:20:19.227653 5801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f369beb03642a5bc095b07cb6b2ece" candidate_uuid: "074cc39baf68454099b816ad22fddc74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef0e719f724454ba531038d0612a66b"
I20260925 15:20:19.227732 5941 raft_consensus.cc:3037] T b4f369beb03642a5bc095b07cb6b2ece P ddb9ca84c5f740ba8983704a4381c165 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:19.227767 5801 raft_consensus.cc:3037] T b4f369beb03642a5bc095b07cb6b2ece P 9ef0e719f724454ba531038d0612a66b [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:19.228556 5941 raft_consensus.cc:2454] T b4f369beb03642a5bc095b07cb6b2ece P ddb9ca84c5f740ba8983704a4381c165 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074cc39baf68454099b816ad22fddc74 in term 1.
I20260925 15:20:19.228740 5801 raft_consensus.cc:2454] T b4f369beb03642a5bc095b07cb6b2ece P 9ef0e719f724454ba531038d0612a66b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074cc39baf68454099b816ad22fddc74 in term 1.
I20260925 15:20:19.228746 6013 leader_election.cc:302] T b4f369beb03642a5bc095b07cb6b2ece P 074cc39baf68454099b816ad22fddc74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074cc39baf68454099b816ad22fddc74, ddb9ca84c5f740ba8983704a4381c165; no voters:
I20260925 15:20:19.228839 6156 raft_consensus.cc:2781] T b4f369beb03642a5bc095b07cb6b2ece P 074cc39baf68454099b816ad22fddc74 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:20:19.228983 6156 raft_consensus.cc:686] T b4f369beb03642a5bc095b07cb6b2ece P 074cc39baf68454099b816ad22fddc74 [term 1 LEADER]: Becoming Leader. State: Replica: 074cc39baf68454099b816ad22fddc74, State: Running, Role: LEADER
I20260925 15:20:19.229099 6156 consensus_queue.cc:237] T b4f369beb03642a5bc095b07cb6b2ece P 074cc39baf68454099b816ad22fddc74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } } peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } }
I20260925 15:20:19.230089 5643 catalog_manager.cc:5495] T b4f369beb03642a5bc095b07cb6b2ece P 074cc39baf68454099b816ad22fddc74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 074cc39baf68454099b816ad22fddc74 (127.3.211.195). New cstate: current_term: 1 leader_uuid: "074cc39baf68454099b816ad22fddc74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } health_report { overall_health: UNKNOWN } } }
Sep 25 15:20:19 dist-test-slave-4bk3 krb5kdc[5599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349618, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.254@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:20:19.273792 6167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:19.265376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:46877 (local address 127.0.0.1:58224)
0925 15:20:19.265542 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:19.265552 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:19.265623 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:19.265867 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:19.265868 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:19.266204 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:19.266253 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:19.266256 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:19.267059 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:19.267061 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:19.267779 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:19.267781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:19.268259 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:19.268261 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:19.268323 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:19.268985 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:19.268991 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:19.270802 (+ 1811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:19.272612 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:19.272621 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:19.272624 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:19.272848 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:19.273102 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:19.273103 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:19.273104 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:19.273178 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:19.273491 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:19.273498 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:19.273543 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 15:20:19.273598 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 15:20:19.273712 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":96,"threads_started":1}
I20260925 15:20:19.273859 5694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:19.265373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58224 (local address 127.3.211.254:46877)
0925 15:20:19.265450 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:19.265452 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:19.265647 (+ 195us) server_negotiation.cc:407] Connection header received
0925 15:20:19.265724 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:19.265725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:19.265744 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:19.265822 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:19.266397 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:19.266901 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:19.267933 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:19.268154 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:19.268235 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:19.270928 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:19.270929 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:19.270929 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:19.270937 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:19.272481 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:19.272941 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:19.272942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:19.272943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:19.273012 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:19.273293 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:19.273294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:19.273295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:19.273423 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:19.273488 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:19.273635 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 15:20:19.273772 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":38}
I20260925 15:20:19.276257 5643 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:58224:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 15:20:19.276566 5643 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:20:19.279074 5921 tablet_service.cc:1467] Processing CreateTablet for tablet ea21fc64988a4961b6c9f5fe04b24b2b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=836aa98015ed4ea2a86982c68939acc9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:20:19.279302 5921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea21fc64988a4961b6c9f5fe04b24b2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:19.279422 6061 tablet_service.cc:1467] Processing CreateTablet for tablet ea21fc64988a4961b6c9f5fe04b24b2b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=836aa98015ed4ea2a86982c68939acc9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:20:19.279408 5781 tablet_service.cc:1467] Processing CreateTablet for tablet ea21fc64988a4961b6c9f5fe04b24b2b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=836aa98015ed4ea2a86982c68939acc9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:20:19.279577 6061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea21fc64988a4961b6c9f5fe04b24b2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:19.279577 5781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea21fc64988a4961b6c9f5fe04b24b2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:19.280741 6151 tablet_bootstrap.cc:492] T ea21fc64988a4961b6c9f5fe04b24b2b P ddb9ca84c5f740ba8983704a4381c165: Bootstrap starting.
I20260925 15:20:19.281103 6152 tablet_bootstrap.cc:492] T ea21fc64988a4961b6c9f5fe04b24b2b P 9ef0e719f724454ba531038d0612a66b: Bootstrap starting.
I20260925 15:20:19.281106 6150 tablet_bootstrap.cc:492] T ea21fc64988a4961b6c9f5fe04b24b2b P 074cc39baf68454099b816ad22fddc74: Bootstrap starting.
I20260925 15:20:19.281200 6151 tablet_bootstrap.cc:654] T ea21fc64988a4961b6c9f5fe04b24b2b P ddb9ca84c5f740ba8983704a4381c165: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:19.281499 6152 tablet_bootstrap.cc:654] T ea21fc64988a4961b6c9f5fe04b24b2b P 9ef0e719f724454ba531038d0612a66b: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:19.281566 6150 tablet_bootstrap.cc:654] T ea21fc64988a4961b6c9f5fe04b24b2b P 074cc39baf68454099b816ad22fddc74: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:19.281920 6151 tablet_bootstrap.cc:492] T ea21fc64988a4961b6c9f5fe04b24b2b P ddb9ca84c5f740ba8983704a4381c165: No bootstrap required, opened a new log
I20260925 15:20:19.281973 6151 ts_tablet_manager.cc:1397] T ea21fc64988a4961b6c9f5fe04b24b2b P ddb9ca84c5f740ba8983704a4381c165: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:19.282193 6150 tablet_bootstrap.cc:492] T ea21fc64988a4961b6c9f5fe04b24b2b P 074cc39baf68454099b816ad22fddc74: No bootstrap required, opened a new log
I20260925 15:20:19.282202 6152 tablet_bootstrap.cc:492] T ea21fc64988a4961b6c9f5fe04b24b2b P 9ef0e719f724454ba531038d0612a66b: No bootstrap required, opened a new log
I20260925 15:20:19.282148 6151 raft_consensus.cc:348] T ea21fc64988a4961b6c9f5fe04b24b2b P ddb9ca84c5f740ba8983704a4381c165 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } } peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } }
I20260925 15:20:19.282236 6152 ts_tablet_manager.cc:1397] T ea21fc64988a4961b6c9f5fe04b24b2b P 9ef0e719f724454ba531038d0612a66b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:19.282236 6151 raft_consensus.cc:374] T ea21fc64988a4961b6c9f5fe04b24b2b P ddb9ca84c5f740ba8983704a4381c165 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:19.282230 6150 ts_tablet_manager.cc:1397] T ea21fc64988a4961b6c9f5fe04b24b2b P 074cc39baf68454099b816ad22fddc74: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260925 15:20:19.282260 6151 raft_consensus.cc:729] T ea21fc64988a4961b6c9f5fe04b24b2b P ddb9ca84c5f740ba8983704a4381c165 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddb9ca84c5f740ba8983704a4381c165, State: Initialized, Role: FOLLOWER
I20260925 15:20:19.282325 6151 consensus_queue.cc:260] T ea21fc64988a4961b6c9f5fe04b24b2b P ddb9ca84c5f740ba8983704a4381c165 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } } peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } }
I20260925 15:20:19.282375 6150 raft_consensus.cc:348] T ea21fc64988a4961b6c9f5fe04b24b2b P 074cc39baf68454099b816ad22fddc74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } } peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } }
I20260925 15:20:19.282421 6151 ts_tablet_manager.cc:1428] T ea21fc64988a4961b6c9f5fe04b24b2b P ddb9ca84c5f740ba8983704a4381c165: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:19.282449 6150 raft_consensus.cc:374] T ea21fc64988a4961b6c9f5fe04b24b2b P 074cc39baf68454099b816ad22fddc74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:19.282380 6152 raft_consensus.cc:348] T ea21fc64988a4961b6c9f5fe04b24b2b P 9ef0e719f724454ba531038d0612a66b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } } peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } }
I20260925 15:20:19.282473 6150 raft_consensus.cc:729] T ea21fc64988a4961b6c9f5fe04b24b2b P 074cc39baf68454099b816ad22fddc74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074cc39baf68454099b816ad22fddc74, State: Initialized, Role: FOLLOWER
I20260925 15:20:19.282480 6152 raft_consensus.cc:374] T ea21fc64988a4961b6c9f5fe04b24b2b P 9ef0e719f724454ba531038d0612a66b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:19.282502 6152 raft_consensus.cc:729] T ea21fc64988a4961b6c9f5fe04b24b2b P 9ef0e719f724454ba531038d0612a66b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef0e719f724454ba531038d0612a66b, State: Initialized, Role: FOLLOWER
I20260925 15:20:19.282471 6157 tablet_replica.cc:439] T ea21fc64988a4961b6c9f5fe04b24b2b P ddb9ca84c5f740ba8983704a4381c165: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } } peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } } }
I20260925 15:20:19.282519 6150 consensus_queue.cc:260] T ea21fc64988a4961b6c9f5fe04b24b2b P 074cc39baf68454099b816ad22fddc74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } } peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } }
I20260925 15:20:19.282542 6152 consensus_queue.cc:260] T ea21fc64988a4961b6c9f5fe04b24b2b P 9ef0e719f724454ba531038d0612a66b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } } peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } }
I20260925 15:20:19.282555 6157 tablet_replica.cc:442] T ea21fc64988a4961b6c9f5fe04b24b2b P ddb9ca84c5f740ba8983704a4381c165: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:20:19.282612 6150 ts_tablet_manager.cc:1428] T ea21fc64988a4961b6c9f5fe04b24b2b P 074cc39baf68454099b816ad22fddc74: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:19.282644 6152 ts_tablet_manager.cc:1428] T ea21fc64988a4961b6c9f5fe04b24b2b P 9ef0e719f724454ba531038d0612a66b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:19.282774 6156 tablet_replica.cc:439] T ea21fc64988a4961b6c9f5fe04b24b2b P 074cc39baf68454099b816ad22fddc74: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } } peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } } }
I20260925 15:20:19.282891 6156 tablet_replica.cc:442] T ea21fc64988a4961b6c9f5fe04b24b2b P 074cc39baf68454099b816ad22fddc74: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:20:19.283017 6158 tablet_replica.cc:439] T ea21fc64988a4961b6c9f5fe04b24b2b P 9ef0e719f724454ba531038d0612a66b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } } peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } } }
I20260925 15:20:19.283131 6158 tablet_replica.cc:442] T ea21fc64988a4961b6c9f5fe04b24b2b P 9ef0e719f724454ba531038d0612a66b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:20:19.343843 6158 raft_consensus.cc:482] T ea21fc64988a4961b6c9f5fe04b24b2b P 9ef0e719f724454ba531038d0612a66b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:20:19.343942 6158 raft_consensus.cc:504] T ea21fc64988a4961b6c9f5fe04b24b2b P 9ef0e719f724454ba531038d0612a66b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } } peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } }
I20260925 15:20:19.344359 6158 leader_election.cc:288] T ea21fc64988a4961b6c9f5fe04b24b2b P 9ef0e719f724454ba531038d0612a66b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddb9ca84c5f740ba8983704a4381c165 (127.3.211.194:35019), 074cc39baf68454099b816ad22fddc74 (127.3.211.195:38695)
I20260925 15:20:19.348460 6145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:19.344589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:44209 (local address 127.3.211.194:35019)
0925 15:20:19.344641 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:19.344644 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:19.344785 (+ 141us) server_negotiation.cc:407] Connection header received
0925 15:20:19.344871 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:19.344872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:19.344883 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:19.344953 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:20:19.345168 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:19.345942 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:19.347539 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:19.348062 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:19.348155 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:19.348165 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:19.348349 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 15:20:19.348393 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 15:20:19.348631 6147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:19.344654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:50493 (local address 127.3.211.195:38695)
0925 15:20:19.344701 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:19.344703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:19.344773 (+ 70us) server_negotiation.cc:407] Connection header received
0925 15:20:19.344882 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:19.344883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:19.344892 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:19.344953 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:20:19.345168 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:19.345940 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:19.347706 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:19.348255 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:19.348376 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:19.348384 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:19.348506 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 15:20:19.348549 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 15:20:19.348517 6171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:19.344542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.194:35019 (local address 127.3.211.193:44209)
0925 15:20:19.344708 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:19.344721 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:19.344790 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:19.344976 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:19.344976 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:19.344979 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:20:19.345096 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:19.345096 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:19.346023 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:19.346025 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:19.347441 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:19.347445 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:19.348186 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:19.348187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:19.348240 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:19.348245 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:20:19.348294 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 15:20:19.348393 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":65,"thread_start_us":96,"threads_started":1}
I20260925 15:20:19.348943 6172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:19.344542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.195:38695 (local address 127.3.211.193:50493)
0925 15:20:19.344708 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:19.344721 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:19.344790 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:19.344975 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:19.344976 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:19.344978 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:20:19.345090 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:19.345093 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:19.346023 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:19.346025 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:19.347622 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:19.347624 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:19.348365 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:19.348365 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:19.348435 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:19.348439 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:20:19.348493 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 15:20:19.348532 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":44,"thread_start_us":98,"threads_started":1}
I20260925 15:20:19.349261 5941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea21fc64988a4961b6c9f5fe04b24b2b" candidate_uuid: "9ef0e719f724454ba531038d0612a66b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddb9ca84c5f740ba8983704a4381c165" is_pre_election: true
I20260925 15:20:19.349272 6081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea21fc64988a4961b6c9f5fe04b24b2b" candidate_uuid: "9ef0e719f724454ba531038d0612a66b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074cc39baf68454099b816ad22fddc74" is_pre_election: true
I20260925 15:20:19.349364 5941 raft_consensus.cc:2454] T ea21fc64988a4961b6c9f5fe04b24b2b P ddb9ca84c5f740ba8983704a4381c165 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ef0e719f724454ba531038d0612a66b in term 0.
I20260925 15:20:19.349376 6081 raft_consensus.cc:2454] T ea21fc64988a4961b6c9f5fe04b24b2b P 074cc39baf68454099b816ad22fddc74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ef0e719f724454ba531038d0612a66b in term 0.
I20260925 15:20:19.349534 5733 leader_election.cc:302] T ea21fc64988a4961b6c9f5fe04b24b2b P 9ef0e719f724454ba531038d0612a66b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ef0e719f724454ba531038d0612a66b, ddb9ca84c5f740ba8983704a4381c165; no voters:
I20260925 15:20:19.349754 6158 raft_consensus.cc:2781] T ea21fc64988a4961b6c9f5fe04b24b2b P 9ef0e719f724454ba531038d0612a66b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:20:19.349795 6158 raft_consensus.cc:482] T ea21fc64988a4961b6c9f5fe04b24b2b P 9ef0e719f724454ba531038d0612a66b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:20:19.349810 6158 raft_consensus.cc:3037] T ea21fc64988a4961b6c9f5fe04b24b2b P 9ef0e719f724454ba531038d0612a66b [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:19.350530 6158 raft_consensus.cc:504] T ea21fc64988a4961b6c9f5fe04b24b2b P 9ef0e719f724454ba531038d0612a66b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } } peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } }
I20260925 15:20:19.350693 6158 leader_election.cc:288] T ea21fc64988a4961b6c9f5fe04b24b2b P 9ef0e719f724454ba531038d0612a66b [CANDIDATE]: Term 1 election: Requested vote from peers ddb9ca84c5f740ba8983704a4381c165 (127.3.211.194:35019), 074cc39baf68454099b816ad22fddc74 (127.3.211.195:38695)
I20260925 15:20:19.350845 5941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea21fc64988a4961b6c9f5fe04b24b2b" candidate_uuid: "9ef0e719f724454ba531038d0612a66b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddb9ca84c5f740ba8983704a4381c165"
I20260925 15:20:19.350845 6081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea21fc64988a4961b6c9f5fe04b24b2b" candidate_uuid: "9ef0e719f724454ba531038d0612a66b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074cc39baf68454099b816ad22fddc74"
I20260925 15:20:19.350920 6081 raft_consensus.cc:3037] T ea21fc64988a4961b6c9f5fe04b24b2b P 074cc39baf68454099b816ad22fddc74 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:19.350943 5941 raft_consensus.cc:3037] T ea21fc64988a4961b6c9f5fe04b24b2b P ddb9ca84c5f740ba8983704a4381c165 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:19.351603 6081 raft_consensus.cc:2454] T ea21fc64988a4961b6c9f5fe04b24b2b P 074cc39baf68454099b816ad22fddc74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ef0e719f724454ba531038d0612a66b in term 1.
I20260925 15:20:19.351603 5941 raft_consensus.cc:2454] T ea21fc64988a4961b6c9f5fe04b24b2b P ddb9ca84c5f740ba8983704a4381c165 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ef0e719f724454ba531038d0612a66b in term 1.
I20260925 15:20:19.351773 5733 leader_election.cc:302] T ea21fc64988a4961b6c9f5fe04b24b2b P 9ef0e719f724454ba531038d0612a66b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ef0e719f724454ba531038d0612a66b, ddb9ca84c5f740ba8983704a4381c165; no voters:
I20260925 15:20:19.351891 6158 raft_consensus.cc:2781] T ea21fc64988a4961b6c9f5fe04b24b2b P 9ef0e719f724454ba531038d0612a66b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:20:19.352026 6158 raft_consensus.cc:686] T ea21fc64988a4961b6c9f5fe04b24b2b P 9ef0e719f724454ba531038d0612a66b [term 1 LEADER]: Becoming Leader. State: Replica: 9ef0e719f724454ba531038d0612a66b, State: Running, Role: LEADER
I20260925 15:20:19.352149 6158 consensus_queue.cc:237] T ea21fc64988a4961b6c9f5fe04b24b2b P 9ef0e719f724454ba531038d0612a66b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } } peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } }
I20260925 15:20:19.352731 6173 tablet_replica.cc:439] T ea21fc64988a4961b6c9f5fe04b24b2b P 9ef0e719f724454ba531038d0612a66b: TxnStatusTablet state changed. Reason: New leader 9ef0e719f724454ba531038d0612a66b. Latest consensus state: current_term: 1 leader_uuid: "9ef0e719f724454ba531038d0612a66b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } } peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } } }
I20260925 15:20:19.352826 6173 tablet_replica.cc:442] T ea21fc64988a4961b6c9f5fe04b24b2b P 9ef0e719f724454ba531038d0612a66b: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:20:19.352996 6175 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 15:20:19.353127 6175 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 15:20:19.353056 5644 catalog_manager.cc:5495] T ea21fc64988a4961b6c9f5fe04b24b2b P 9ef0e719f724454ba531038d0612a66b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ef0e719f724454ba531038d0612a66b (127.3.211.193). New cstate: current_term: 1 leader_uuid: "9ef0e719f724454ba531038d0612a66b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } health_report { overall_health: UNKNOWN } } }
I20260925 15:20:19.362519 6167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:19.359889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:34301 (local address 127.0.0.1:52382)
0925 15:20:19.359969 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:19.359981 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:19.360034 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:19.360234 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:19.360235 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:19.360236 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:20:19.360289 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:19.360292 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:19.360980 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:19.360982 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:19.361560 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:19.361562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:19.361961 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:19.361962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:19.362026 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:19.362046 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:20:19.362331 (+ 285us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:20:19.362341 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 15:20:19.362400 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 15:20:19.362451 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 15:20:19.362510 6149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:19.359981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52382 (local address 127.3.211.193:34301)
0925 15:20:19.360026 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:19.360028 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:19.360037 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:20:19.360117 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:19.360118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:19.360138 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:19.360228 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:20:19.360376 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:19.360889 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:19.361655 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:19.361850 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:19.361961 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:19.362157 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:20:19.362261 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:20:19.362353 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:19.362405 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 15:20:19.362427 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 15:20:19.364424 6081 raft_consensus.cc:1261] T ea21fc64988a4961b6c9f5fe04b24b2b P 074cc39baf68454099b816ad22fddc74 [term 1 FOLLOWER]: Refusing update from remote peer 9ef0e719f724454ba531038d0612a66b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:20:19.364414 5941 raft_consensus.cc:1261] T ea21fc64988a4961b6c9f5fe04b24b2b P ddb9ca84c5f740ba8983704a4381c165 [term 1 FOLLOWER]: Refusing update from remote peer 9ef0e719f724454ba531038d0612a66b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:20:19.364778 6173 consensus_queue.cc:1035] T ea21fc64988a4961b6c9f5fe04b24b2b P 9ef0e719f724454ba531038d0612a66b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:20:19.364975 6158 consensus_queue.cc:1035] T ea21fc64988a4961b6c9f5fe04b24b2b P 9ef0e719f724454ba531038d0612a66b [LEADER]: Connected to new peer: Peer: permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:20:19.366045 6157 tablet_replica.cc:439] T ea21fc64988a4961b6c9f5fe04b24b2b P ddb9ca84c5f740ba8983704a4381c165: TxnStatusTablet state changed. Reason: New leader 9ef0e719f724454ba531038d0612a66b. Latest consensus state: current_term: 1 leader_uuid: "9ef0e719f724454ba531038d0612a66b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } } peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } } }
I20260925 15:20:19.366139 6157 tablet_replica.cc:442] T ea21fc64988a4961b6c9f5fe04b24b2b P ddb9ca84c5f740ba8983704a4381c165: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:20:19.366111 6156 tablet_replica.cc:439] T ea21fc64988a4961b6c9f5fe04b24b2b P 074cc39baf68454099b816ad22fddc74: TxnStatusTablet state changed. Reason: New leader 9ef0e719f724454ba531038d0612a66b. Latest consensus state: current_term: 1 leader_uuid: "9ef0e719f724454ba531038d0612a66b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } } peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } } }
I20260925 15:20:19.366195 6156 tablet_replica.cc:442] T ea21fc64988a4961b6c9f5fe04b24b2b P 074cc39baf68454099b816ad22fddc74: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:20:19.367118 6173 tablet_replica.cc:439] T ea21fc64988a4961b6c9f5fe04b24b2b P 9ef0e719f724454ba531038d0612a66b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9ef0e719f724454ba531038d0612a66b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } } peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } } }
I20260925 15:20:19.367214 6173 tablet_replica.cc:442] T ea21fc64988a4961b6c9f5fe04b24b2b P 9ef0e719f724454ba531038d0612a66b: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:20:19.367264 6156 tablet_replica.cc:439] T ea21fc64988a4961b6c9f5fe04b24b2b P 074cc39baf68454099b816ad22fddc74: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9ef0e719f724454ba531038d0612a66b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } } peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } } }
I20260925 15:20:19.367348 6156 tablet_replica.cc:442] T ea21fc64988a4961b6c9f5fe04b24b2b P 074cc39baf68454099b816ad22fddc74: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:20:19.367576 6180 mvcc.cc:205] Tried to move back new op lower bound from 7333272040913870848 to 7333272040867172352. Current Snapshot: MvccSnapshot[applied={T|T < 7333272040913870848}]
I20260925 15:20:19.368494 6158 tablet_replica.cc:439] T ea21fc64988a4961b6c9f5fe04b24b2b P 9ef0e719f724454ba531038d0612a66b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9ef0e719f724454ba531038d0612a66b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } } peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } } }
I20260925 15:20:19.368578 6158 tablet_replica.cc:442] T ea21fc64988a4961b6c9f5fe04b24b2b P 9ef0e719f724454ba531038d0612a66b: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:20:19.369083 6157 tablet_replica.cc:439] T ea21fc64988a4961b6c9f5fe04b24b2b P ddb9ca84c5f740ba8983704a4381c165: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9ef0e719f724454ba531038d0612a66b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } } peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } } }
I20260925 15:20:19.369184 6157 tablet_replica.cc:442] T ea21fc64988a4961b6c9f5fe04b24b2b P ddb9ca84c5f740ba8983704a4381c165: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:20:19.369388 6179 mvcc.cc:205] Tried to move back new op lower bound from 7333272040913870848 to 7333272040867172352. Current Snapshot: MvccSnapshot[applied={T|T < 7333272040913870848}]
I20260925 15:20:19.373382 6147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:19.370415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59112 (local address 127.3.211.195:38695)
0925 15:20:19.370455 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:19.370457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:19.370467 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:20:19.370506 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:19.370506 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:19.370525 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:19.370603 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:20:19.370808 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:19.371564 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:19.372424 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:19.372744 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:19.372817 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:19.372980 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:20:19.373113 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:20:19.373197 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:19.373277 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 15:20:19.373300 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 15:20:19.379895 6133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:19.376990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:34301 (local address 127.3.211.195:58803)
0925 15:20:19.377095 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:19.377111 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:19.377182 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:19.377319 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:19.377319 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:19.377321 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:20:19.377388 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:19.377389 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:19.378319 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:19.378321 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:19.378958 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:19.378960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:19.379456 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:19.379457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:19.379510 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:19.379529 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:20:19.379747 (+ 218us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:20:19.379748 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 15:20:19.379788 (+ 40us) client_negotiation.cc:240] Negotiation successful
0925 15:20:19.379826 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 15:20:19.379902 6149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:19.377059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:58803 (local address 127.3.211.193:34301)
0925 15:20:19.377102 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:19.377104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:19.377192 (+ 88us) server_negotiation.cc:407] Connection header received
0925 15:20:19.377242 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:19.377243 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:19.377253 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:19.377307 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:20:19.377458 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:19.378208 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:19.379060 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:19.379373 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:19.379447 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:19.379641 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:20:19.379677 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:20:19.379738 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:19.379796 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 15:20:19.379817 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 15:20:19.382920 5941 raft_consensus.cc:1261] T b4f369beb03642a5bc095b07cb6b2ece P ddb9ca84c5f740ba8983704a4381c165 [term 1 FOLLOWER]: Refusing update from remote peer 074cc39baf68454099b816ad22fddc74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:20:19.383013 5801 raft_consensus.cc:1261] T b4f369beb03642a5bc095b07cb6b2ece P 9ef0e719f724454ba531038d0612a66b [term 1 FOLLOWER]: Refusing update from remote peer 074cc39baf68454099b816ad22fddc74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:20:19.383158 6156 consensus_queue.cc:1035] T b4f369beb03642a5bc095b07cb6b2ece P 074cc39baf68454099b816ad22fddc74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:20:19.383287 6160 consensus_queue.cc:1035] T b4f369beb03642a5bc095b07cb6b2ece P 074cc39baf68454099b816ad22fddc74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:20:19.385398 6178 mvcc.cc:205] Tried to move back new op lower bound from 7333272040990683136 to 7333272040363089920. Current Snapshot: MvccSnapshot[applied={T|T < 7333272040990683136}]
I20260925 15:20:19.393363 6147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:19.390200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:49837 (local address 127.3.211.195:38695)
0925 15:20:19.390254 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:19.390256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:19.390487 (+ 231us) server_negotiation.cc:407] Connection header received
0925 15:20:19.390518 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:19.390518 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:19.390537 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:19.390589 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:20:19.390721 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:19.391186 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:19.392453 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:19.392762 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:19.392805 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:19.393076 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:20:19.393110 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:20:19.393161 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:19.393223 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 15:20:19.393285 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 15:20:19.393376 6194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:19.390150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.195:38695 (local address 127.3.211.193:49837)
0925 15:20:19.390400 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:19.390412 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:19.390471 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:19.390592 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:19.390593 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:19.390595 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:20:19.390662 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:19.390663 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:19.391263 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:19.391264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:19.392316 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:19.392319 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:19.392864 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:19.392865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:19.392933 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:19.392954 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:20:19.393165 (+ 211us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:20:19.393167 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 15:20:19.393219 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 15:20:19.393268 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":114,"threads_started":1}
I20260925 15:20:19.403254 5644 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:20:19.403383 5644 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:20:19.404170 5644 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bba60aadb5c245e6bb6d4680cdfa0adc: Sending DeleteTablet for 3 replicas of tablet b4f369beb03642a5bc095b07cb6b2ece
I20260925 15:20:19.404518 5921 tablet_service.cc:1514] Processing DeleteTablet for tablet b4f369beb03642a5bc095b07cb6b2ece with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:20:19 UTC) from {username='kudu', principal='kudu/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:47456
I20260925 15:20:19.404623 5781 tablet_service.cc:1514] Processing DeleteTablet for tablet b4f369beb03642a5bc095b07cb6b2ece with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:20:19 UTC) from {username='kudu', principal='kudu/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:52376
I20260925 15:20:19.404808 5644 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58216:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:20:19.404958 5644 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:20:19.405184 6061 tablet_service.cc:1514] Processing DeleteTablet for tablet b4f369beb03642a5bc095b07cb6b2ece with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:20:19 UTC) from {username='kudu', principal='kudu/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:59108
I20260925 15:20:19.405535 6198 tablet_replica.cc:330] T b4f369beb03642a5bc095b07cb6b2ece P 9ef0e719f724454ba531038d0612a66b: stopping tablet replica
I20260925 15:20:19.405664 6199 tablet_replica.cc:330] T b4f369beb03642a5bc095b07cb6b2ece P 074cc39baf68454099b816ad22fddc74: stopping tablet replica
I20260925 15:20:19.405802 6199 raft_consensus.cc:2229] T b4f369beb03642a5bc095b07cb6b2ece P 074cc39baf68454099b816ad22fddc74 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:20:19.405957 6198 raft_consensus.cc:2229] T b4f369beb03642a5bc095b07cb6b2ece P 9ef0e719f724454ba531038d0612a66b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:20:19.406054 6061 tablet_service.cc:1467] Processing CreateTablet for tablet ded91d524fa54bf4b837d4aeb64a4ef9 (DEFAULT_TABLE table=test-table [id=544ffca789b7482ea9bbfb89cb5a8180]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:20:19.406404 6061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ded91d524fa54bf4b837d4aeb64a4ef9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:19.406417 5921 tablet_service.cc:1467] Processing CreateTablet for tablet ded91d524fa54bf4b837d4aeb64a4ef9 (DEFAULT_TABLE table=test-table [id=544ffca789b7482ea9bbfb89cb5a8180]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:20:19.406556 5921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ded91d524fa54bf4b837d4aeb64a4ef9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:19.407565 6151 tablet_bootstrap.cc:492] T ded91d524fa54bf4b837d4aeb64a4ef9 P ddb9ca84c5f740ba8983704a4381c165: Bootstrap starting.
I20260925 15:20:19.407824 6150 tablet_bootstrap.cc:492] T ded91d524fa54bf4b837d4aeb64a4ef9 P 074cc39baf68454099b816ad22fddc74: Bootstrap starting.
I20260925 15:20:19.408007 6151 tablet_bootstrap.cc:654] T ded91d524fa54bf4b837d4aeb64a4ef9 P ddb9ca84c5f740ba8983704a4381c165: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:19.408208 6150 tablet_bootstrap.cc:654] T ded91d524fa54bf4b837d4aeb64a4ef9 P 074cc39baf68454099b816ad22fddc74: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:19.408882 6199 raft_consensus.cc:2258] T b4f369beb03642a5bc095b07cb6b2ece P 074cc39baf68454099b816ad22fddc74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:20:19.406258 5781 tablet_service.cc:1467] Processing CreateTablet for tablet ded91d524fa54bf4b837d4aeb64a4ef9 (DEFAULT_TABLE table=test-table [id=544ffca789b7482ea9bbfb89cb5a8180]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:20:19.409101 5781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ded91d524fa54bf4b837d4aeb64a4ef9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:19.410530 6152 tablet_bootstrap.cc:492] T ded91d524fa54bf4b837d4aeb64a4ef9 P 9ef0e719f724454ba531038d0612a66b: Bootstrap starting.
I20260925 15:20:19.411159 6152 tablet_bootstrap.cc:654] T ded91d524fa54bf4b837d4aeb64a4ef9 P 9ef0e719f724454ba531038d0612a66b: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:19.411667 6198 raft_consensus.cc:2258] T b4f369beb03642a5bc095b07cb6b2ece P 9ef0e719f724454ba531038d0612a66b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:20:19.411913 6152 tablet_bootstrap.cc:492] T ded91d524fa54bf4b837d4aeb64a4ef9 P 9ef0e719f724454ba531038d0612a66b: No bootstrap required, opened a new log
I20260925 15:20:19.411960 6152 ts_tablet_manager.cc:1397] T ded91d524fa54bf4b837d4aeb64a4ef9 P 9ef0e719f724454ba531038d0612a66b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:19.412118 6152 raft_consensus.cc:348] T ded91d524fa54bf4b837d4aeb64a4ef9 P 9ef0e719f724454ba531038d0612a66b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } } peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } }
I20260925 15:20:19.412186 6152 raft_consensus.cc:374] T ded91d524fa54bf4b837d4aeb64a4ef9 P 9ef0e719f724454ba531038d0612a66b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:19.412206 6152 raft_consensus.cc:729] T ded91d524fa54bf4b837d4aeb64a4ef9 P 9ef0e719f724454ba531038d0612a66b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef0e719f724454ba531038d0612a66b, State: Initialized, Role: FOLLOWER
I20260925 15:20:19.412248 6152 consensus_queue.cc:260] T ded91d524fa54bf4b837d4aeb64a4ef9 P 9ef0e719f724454ba531038d0612a66b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } } peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } }
I20260925 15:20:19.412279 6199 ts_tablet_manager.cc:1905] T b4f369beb03642a5bc095b07cb6b2ece P 074cc39baf68454099b816ad22fddc74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:20:19.412355 6152 ts_tablet_manager.cc:1428] T ded91d524fa54bf4b837d4aeb64a4ef9 P 9ef0e719f724454ba531038d0612a66b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:19.413223 6198 ts_tablet_manager.cc:1905] T b4f369beb03642a5bc095b07cb6b2ece P 9ef0e719f724454ba531038d0612a66b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:20:19.415030 6198 ts_tablet_manager.cc:1918] T b4f369beb03642a5bc095b07cb6b2ece P 9ef0e719f724454ba531038d0612a66b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:20:19.415210 6198 log.cc:1196] T b4f369beb03642a5bc095b07cb6b2ece P 9ef0e719f724454ba531038d0612a66b: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-0/wal/wals/b4f369beb03642a5bc095b07cb6b2ece
I20260925 15:20:19.415934 6198 ts_tablet_manager.cc:1939] T b4f369beb03642a5bc095b07cb6b2ece P 9ef0e719f724454ba531038d0612a66b: Deleting consensus metadata
I20260925 15:20:19.416216 6199 ts_tablet_manager.cc:1918] T b4f369beb03642a5bc095b07cb6b2ece P 074cc39baf68454099b816ad22fddc74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:20:19.416296 6199 log.cc:1196] T b4f369beb03642a5bc095b07cb6b2ece P 074cc39baf68454099b816ad22fddc74: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-2/wal/wals/b4f369beb03642a5bc095b07cb6b2ece
I20260925 15:20:19.416771 5628 catalog_manager.cc:4841] TS 9ef0e719f724454ba531038d0612a66b (127.3.211.193:34301): tablet b4f369beb03642a5bc095b07cb6b2ece (table test-table [id=77726c28fc6a4a83afc4961448849e97]) successfully deleted
I20260925 15:20:19.416851 6197 tablet_replica.cc:330] T b4f369beb03642a5bc095b07cb6b2ece P ddb9ca84c5f740ba8983704a4381c165: stopping tablet replica
I20260925 15:20:19.416906 6150 tablet_bootstrap.cc:492] T ded91d524fa54bf4b837d4aeb64a4ef9 P 074cc39baf68454099b816ad22fddc74: No bootstrap required, opened a new log
I20260925 15:20:19.416952 6150 ts_tablet_manager.cc:1397] T ded91d524fa54bf4b837d4aeb64a4ef9 P 074cc39baf68454099b816ad22fddc74: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.001s
I20260925 15:20:19.416983 6197 raft_consensus.cc:2229] T b4f369beb03642a5bc095b07cb6b2ece P ddb9ca84c5f740ba8983704a4381c165 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:20:19.417070 6197 raft_consensus.cc:2258] T b4f369beb03642a5bc095b07cb6b2ece P ddb9ca84c5f740ba8983704a4381c165 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:20:19.417109 6150 raft_consensus.cc:348] T ded91d524fa54bf4b837d4aeb64a4ef9 P 074cc39baf68454099b816ad22fddc74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } } peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } }
I20260925 15:20:19.417194 6150 raft_consensus.cc:374] T ded91d524fa54bf4b837d4aeb64a4ef9 P 074cc39baf68454099b816ad22fddc74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:19.417220 6150 raft_consensus.cc:729] T ded91d524fa54bf4b837d4aeb64a4ef9 P 074cc39baf68454099b816ad22fddc74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074cc39baf68454099b816ad22fddc74, State: Initialized, Role: FOLLOWER
I20260925 15:20:19.417269 6150 consensus_queue.cc:260] T ded91d524fa54bf4b837d4aeb64a4ef9 P 074cc39baf68454099b816ad22fddc74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } } peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } }
I20260925 15:20:19.417335 6199 ts_tablet_manager.cc:1939] T b4f369beb03642a5bc095b07cb6b2ece P 074cc39baf68454099b816ad22fddc74: Deleting consensus metadata
I20260925 15:20:19.417388 6150 ts_tablet_manager.cc:1428] T ded91d524fa54bf4b837d4aeb64a4ef9 P 074cc39baf68454099b816ad22fddc74: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:19.417872 5631 catalog_manager.cc:4841] TS 074cc39baf68454099b816ad22fddc74 (127.3.211.195:38695): tablet b4f369beb03642a5bc095b07cb6b2ece (table test-table [id=77726c28fc6a4a83afc4961448849e97]) successfully deleted
I20260925 15:20:19.418207 6197 ts_tablet_manager.cc:1905] T b4f369beb03642a5bc095b07cb6b2ece P ddb9ca84c5f740ba8983704a4381c165: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:20:19.420049 6197 ts_tablet_manager.cc:1918] T b4f369beb03642a5bc095b07cb6b2ece P ddb9ca84c5f740ba8983704a4381c165: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:20:19.420130 6197 log.cc:1196] T b4f369beb03642a5bc095b07cb6b2ece P ddb9ca84c5f740ba8983704a4381c165: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-1/wal/wals/b4f369beb03642a5bc095b07cb6b2ece
I20260925 15:20:19.420480 6197 ts_tablet_manager.cc:1939] T b4f369beb03642a5bc095b07cb6b2ece P ddb9ca84c5f740ba8983704a4381c165: Deleting consensus metadata
I20260925 15:20:19.420959 5629 catalog_manager.cc:4841] TS ddb9ca84c5f740ba8983704a4381c165 (127.3.211.194:35019): tablet b4f369beb03642a5bc095b07cb6b2ece (table test-table [id=77726c28fc6a4a83afc4961448849e97]) successfully deleted
I20260925 15:20:19.421231 6151 tablet_bootstrap.cc:492] T ded91d524fa54bf4b837d4aeb64a4ef9 P ddb9ca84c5f740ba8983704a4381c165: No bootstrap required, opened a new log
I20260925 15:20:19.421273 6151 ts_tablet_manager.cc:1397] T ded91d524fa54bf4b837d4aeb64a4ef9 P ddb9ca84c5f740ba8983704a4381c165: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.000s
I20260925 15:20:19.421424 6151 raft_consensus.cc:348] T ded91d524fa54bf4b837d4aeb64a4ef9 P ddb9ca84c5f740ba8983704a4381c165 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } } peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } }
I20260925 15:20:19.421494 6151 raft_consensus.cc:374] T ded91d524fa54bf4b837d4aeb64a4ef9 P ddb9ca84c5f740ba8983704a4381c165 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:19.421514 6151 raft_consensus.cc:729] T ded91d524fa54bf4b837d4aeb64a4ef9 P ddb9ca84c5f740ba8983704a4381c165 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddb9ca84c5f740ba8983704a4381c165, State: Initialized, Role: FOLLOWER
I20260925 15:20:19.421558 6151 consensus_queue.cc:260] T ded91d524fa54bf4b837d4aeb64a4ef9 P ddb9ca84c5f740ba8983704a4381c165 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } } peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } }
I20260925 15:20:19.421655 6151 ts_tablet_manager.cc:1428] T ded91d524fa54bf4b837d4aeb64a4ef9 P ddb9ca84c5f740ba8983704a4381c165: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 15:20:19.604272 6173 raft_consensus.cc:482] T ded91d524fa54bf4b837d4aeb64a4ef9 P 9ef0e719f724454ba531038d0612a66b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:20:19.604454 6173 raft_consensus.cc:504] T ded91d524fa54bf4b837d4aeb64a4ef9 P 9ef0e719f724454ba531038d0612a66b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } } peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } }
I20260925 15:20:19.604692 6173 leader_election.cc:288] T ded91d524fa54bf4b837d4aeb64a4ef9 P 9ef0e719f724454ba531038d0612a66b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddb9ca84c5f740ba8983704a4381c165 (127.3.211.194:35019), 074cc39baf68454099b816ad22fddc74 (127.3.211.195:38695)
I20260925 15:20:19.604945 5941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ded91d524fa54bf4b837d4aeb64a4ef9" candidate_uuid: "9ef0e719f724454ba531038d0612a66b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddb9ca84c5f740ba8983704a4381c165" is_pre_election: true
I20260925 15:20:19.604969 6081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ded91d524fa54bf4b837d4aeb64a4ef9" candidate_uuid: "9ef0e719f724454ba531038d0612a66b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074cc39baf68454099b816ad22fddc74" is_pre_election: true
I20260925 15:20:19.605064 5941 raft_consensus.cc:2454] T ded91d524fa54bf4b837d4aeb64a4ef9 P ddb9ca84c5f740ba8983704a4381c165 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ef0e719f724454ba531038d0612a66b in term 0.
I20260925 15:20:19.605067 6081 raft_consensus.cc:2454] T ded91d524fa54bf4b837d4aeb64a4ef9 P 074cc39baf68454099b816ad22fddc74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ef0e719f724454ba531038d0612a66b in term 0.
I20260925 15:20:19.605262 5735 leader_election.cc:302] T ded91d524fa54bf4b837d4aeb64a4ef9 P 9ef0e719f724454ba531038d0612a66b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074cc39baf68454099b816ad22fddc74, 9ef0e719f724454ba531038d0612a66b; no voters:
I20260925 15:20:19.605376 6173 raft_consensus.cc:2781] T ded91d524fa54bf4b837d4aeb64a4ef9 P 9ef0e719f724454ba531038d0612a66b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:20:19.605427 6173 raft_consensus.cc:482] T ded91d524fa54bf4b837d4aeb64a4ef9 P 9ef0e719f724454ba531038d0612a66b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:20:19.605455 6173 raft_consensus.cc:3037] T ded91d524fa54bf4b837d4aeb64a4ef9 P 9ef0e719f724454ba531038d0612a66b [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:19.606173 6173 raft_consensus.cc:504] T ded91d524fa54bf4b837d4aeb64a4ef9 P 9ef0e719f724454ba531038d0612a66b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } } peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } }
I20260925 15:20:19.606339 6173 leader_election.cc:288] T ded91d524fa54bf4b837d4aeb64a4ef9 P 9ef0e719f724454ba531038d0612a66b [CANDIDATE]: Term 1 election: Requested vote from peers ddb9ca84c5f740ba8983704a4381c165 (127.3.211.194:35019), 074cc39baf68454099b816ad22fddc74 (127.3.211.195:38695)
I20260925 15:20:19.606563 5941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ded91d524fa54bf4b837d4aeb64a4ef9" candidate_uuid: "9ef0e719f724454ba531038d0612a66b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddb9ca84c5f740ba8983704a4381c165"
I20260925 15:20:19.606554 6081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ded91d524fa54bf4b837d4aeb64a4ef9" candidate_uuid: "9ef0e719f724454ba531038d0612a66b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074cc39baf68454099b816ad22fddc74"
I20260925 15:20:19.606643 6081 raft_consensus.cc:3037] T ded91d524fa54bf4b837d4aeb64a4ef9 P 074cc39baf68454099b816ad22fddc74 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:19.606643 5941 raft_consensus.cc:3037] T ded91d524fa54bf4b837d4aeb64a4ef9 P ddb9ca84c5f740ba8983704a4381c165 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:19.607405 5941 raft_consensus.cc:2454] T ded91d524fa54bf4b837d4aeb64a4ef9 P ddb9ca84c5f740ba8983704a4381c165 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ef0e719f724454ba531038d0612a66b in term 1.
I20260925 15:20:19.607405 6081 raft_consensus.cc:2454] T ded91d524fa54bf4b837d4aeb64a4ef9 P 074cc39baf68454099b816ad22fddc74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ef0e719f724454ba531038d0612a66b in term 1.
I20260925 15:20:19.607626 5733 leader_election.cc:302] T ded91d524fa54bf4b837d4aeb64a4ef9 P 9ef0e719f724454ba531038d0612a66b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ef0e719f724454ba531038d0612a66b, ddb9ca84c5f740ba8983704a4381c165; no voters:
I20260925 15:20:19.607741 6173 raft_consensus.cc:2781] T ded91d524fa54bf4b837d4aeb64a4ef9 P 9ef0e719f724454ba531038d0612a66b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:20:19.607795 6173 raft_consensus.cc:686] T ded91d524fa54bf4b837d4aeb64a4ef9 P 9ef0e719f724454ba531038d0612a66b [term 1 LEADER]: Becoming Leader. State: Replica: 9ef0e719f724454ba531038d0612a66b, State: Running, Role: LEADER
I20260925 15:20:19.607872 6173 consensus_queue.cc:237] T ded91d524fa54bf4b837d4aeb64a4ef9 P 9ef0e719f724454ba531038d0612a66b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } } peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } }
I20260925 15:20:19.608525 5643 catalog_manager.cc:5495] T ded91d524fa54bf4b837d4aeb64a4ef9 P 9ef0e719f724454ba531038d0612a66b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ef0e719f724454ba531038d0612a66b (127.3.211.193). New cstate: current_term: 1 leader_uuid: "9ef0e719f724454ba531038d0612a66b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } health_report { overall_health: UNKNOWN } } }
I20260925 15:20:19.641582 6149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:19.638886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52388 (local address 127.3.211.193:34301)
0925 15:20:19.638958 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:19.638960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:19.638970 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:20:19.638995 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:19.638996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:19.639011 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:19.639078 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:20:19.639181 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:19.639985 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:19.640811 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:19.641145 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:19.641217 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:19.641315 (+ 98us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:20:19.641351 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:20:19.641404 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:19.641453 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 15:20:19.641478 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 15:20:19.647420 6149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:19.644125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:52057 (local address 127.3.211.193:34301)
0925 15:20:19.644184 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:19.644185 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:19.644196 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:20:19.644219 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:19.644219 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:19.644234 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:19.644280 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:20:19.644429 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:19.645201 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:19.646436 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:19.646799 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:19.646877 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:19.647066 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:20:19.647096 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:20:19.647149 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:19.647315 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 15:20:19.647344 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 15:20:19.647421 6194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:19.644070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:34301 (local address 127.3.211.193:52057)
0925 15:20:19.644123 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:19.644133 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:19.644173 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:19.644286 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:19.644286 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:19.644288 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:20:19.644357 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:19.644359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:19.645340 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:19.645341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:19.646268 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:19.646271 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:19.646911 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:19.646911 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:19.646966 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:19.646973 (+ 7us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:20:19.647179 (+ 206us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:20:19.647181 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 15:20:19.647240 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 15:20:19.647300 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 15:20:19.649847 6081 raft_consensus.cc:1261] T ded91d524fa54bf4b837d4aeb64a4ef9 P 074cc39baf68454099b816ad22fddc74 [term 1 FOLLOWER]: Refusing update from remote peer 9ef0e719f724454ba531038d0612a66b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:20:19.649824 5941 raft_consensus.cc:1261] T ded91d524fa54bf4b837d4aeb64a4ef9 P ddb9ca84c5f740ba8983704a4381c165 [term 1 FOLLOWER]: Refusing update from remote peer 9ef0e719f724454ba531038d0612a66b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:20:19.650038 6185 consensus_queue.cc:1035] T ded91d524fa54bf4b837d4aeb64a4ef9 P 9ef0e719f724454ba531038d0612a66b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:20:19.650134 6173 consensus_queue.cc:1035] T ded91d524fa54bf4b837d4aeb64a4ef9 P 9ef0e719f724454ba531038d0612a66b [LEADER]: Connected to new peer: Peer: permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 15:20:19.656069 6178 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 15:20:21.186982 5644 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:20:21.187098 5644 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:20:21.188495 5644 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bba60aadb5c245e6bb6d4680cdfa0adc: Sending DeleteTablet for 3 replicas of tablet ded91d524fa54bf4b837d4aeb64a4ef9
I20260925 15:20:21.188800 5780 tablet_service.cc:1514] Processing DeleteTablet for tablet ded91d524fa54bf4b837d4aeb64a4ef9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:20:21 UTC) from {username='kudu', principal='kudu/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:52376
I20260925 15:20:21.188897 5921 tablet_service.cc:1514] Processing DeleteTablet for tablet ded91d524fa54bf4b837d4aeb64a4ef9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:20:21 UTC) from {username='kudu', principal='kudu/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:47456
I20260925 15:20:21.188897 6061 tablet_service.cc:1514] Processing DeleteTablet for tablet ded91d524fa54bf4b837d4aeb64a4ef9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:20:21 UTC) from {username='kudu', principal='kudu/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:59108
I20260925 15:20:21.189148 6228 tablet_replica.cc:330] T ded91d524fa54bf4b837d4aeb64a4ef9 P 074cc39baf68454099b816ad22fddc74: stopping tablet replica
I20260925 15:20:21.189213 6228 raft_consensus.cc:2229] T ded91d524fa54bf4b837d4aeb64a4ef9 P 074cc39baf68454099b816ad22fddc74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:20:21.189288 6228 raft_consensus.cc:2258] T ded91d524fa54bf4b837d4aeb64a4ef9 P 074cc39baf68454099b816ad22fddc74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:20:21.189399 6227 tablet_replica.cc:330] T ded91d524fa54bf4b837d4aeb64a4ef9 P 9ef0e719f724454ba531038d0612a66b: stopping tablet replica
I20260925 15:20:21.189461 6227 raft_consensus.cc:2229] T ded91d524fa54bf4b837d4aeb64a4ef9 P 9ef0e719f724454ba531038d0612a66b [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:20:21.189575 6227 raft_consensus.cc:2258] T ded91d524fa54bf4b837d4aeb64a4ef9 P 9ef0e719f724454ba531038d0612a66b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:20:21.189579 6228 ts_tablet_manager.cc:1905] T ded91d524fa54bf4b837d4aeb64a4ef9 P 074cc39baf68454099b816ad22fddc74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:20:21.189714 6229 tablet_replica.cc:330] T ded91d524fa54bf4b837d4aeb64a4ef9 P ddb9ca84c5f740ba8983704a4381c165: stopping tablet replica
I20260925 15:20:21.189777 6229 raft_consensus.cc:2229] T ded91d524fa54bf4b837d4aeb64a4ef9 P ddb9ca84c5f740ba8983704a4381c165 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:20:21.189824 6229 raft_consensus.cc:2258] T ded91d524fa54bf4b837d4aeb64a4ef9 P ddb9ca84c5f740ba8983704a4381c165 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:20:21.190053 6229 ts_tablet_manager.cc:1905] T ded91d524fa54bf4b837d4aeb64a4ef9 P ddb9ca84c5f740ba8983704a4381c165: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:20:21.191376 6228 ts_tablet_manager.cc:1918] T ded91d524fa54bf4b837d4aeb64a4ef9 P 074cc39baf68454099b816ad22fddc74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 15:20:21.191447 6228 log.cc:1196] T ded91d524fa54bf4b837d4aeb64a4ef9 P 074cc39baf68454099b816ad22fddc74: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-2/wal/wals/ded91d524fa54bf4b837d4aeb64a4ef9
I20260925 15:20:21.191680 6228 ts_tablet_manager.cc:1939] T ded91d524fa54bf4b837d4aeb64a4ef9 P 074cc39baf68454099b816ad22fddc74: Deleting consensus metadata
I20260925 15:20:21.191970 6227 ts_tablet_manager.cc:1905] T ded91d524fa54bf4b837d4aeb64a4ef9 P 9ef0e719f724454ba531038d0612a66b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:20:21.192088 5631 catalog_manager.cc:4841] TS 074cc39baf68454099b816ad22fddc74 (127.3.211.195:38695): tablet ded91d524fa54bf4b837d4aeb64a4ef9 (table test-table [id=544ffca789b7482ea9bbfb89cb5a8180]) successfully deleted
I20260925 15:20:21.192396 6229 ts_tablet_manager.cc:1918] T ded91d524fa54bf4b837d4aeb64a4ef9 P ddb9ca84c5f740ba8983704a4381c165: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 15:20:21.192463 6229 log.cc:1196] T ded91d524fa54bf4b837d4aeb64a4ef9 P ddb9ca84c5f740ba8983704a4381c165: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-1/wal/wals/ded91d524fa54bf4b837d4aeb64a4ef9
I20260925 15:20:21.192685 6229 ts_tablet_manager.cc:1939] T ded91d524fa54bf4b837d4aeb64a4ef9 P ddb9ca84c5f740ba8983704a4381c165: Deleting consensus metadata
I20260925 15:20:21.193017 5629 catalog_manager.cc:4841] TS ddb9ca84c5f740ba8983704a4381c165 (127.3.211.194:35019): tablet ded91d524fa54bf4b837d4aeb64a4ef9 (table test-table [id=544ffca789b7482ea9bbfb89cb5a8180]) successfully deleted
I20260925 15:20:21.193833 6227 ts_tablet_manager.cc:1918] T ded91d524fa54bf4b837d4aeb64a4ef9 P 9ef0e719f724454ba531038d0612a66b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 15:20:21.193886 6227 log.cc:1196] T ded91d524fa54bf4b837d4aeb64a4ef9 P 9ef0e719f724454ba531038d0612a66b: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-0/wal/wals/ded91d524fa54bf4b837d4aeb64a4ef9
I20260925 15:20:21.194070 6227 ts_tablet_manager.cc:1939] T ded91d524fa54bf4b837d4aeb64a4ef9 P 9ef0e719f724454ba531038d0612a66b: Deleting consensus metadata
I20260925 15:20:21.194387 5628 catalog_manager.cc:4841] TS 9ef0e719f724454ba531038d0612a66b (127.3.211.193:34301): tablet ded91d524fa54bf4b837d4aeb64a4ef9 (table test-table [id=544ffca789b7482ea9bbfb89cb5a8180]) successfully deleted
Sep 25 15:20:21 dist-test-slave-4bk3 krb5kdc[5599](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349621, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 15:20:21 dist-test-slave-4bk3 krb5kdc[5599](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349621, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:20:21.216310 6237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:21.206894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58228 (local address 127.3.211.254:46877)
0925 15:20:21.207062 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:21.207064 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:21.207075 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:20:21.207105 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:21.207106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:21.207116 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:21.207194 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:21.207888 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:21.208782 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:21.209685 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:21.209914 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:21.209997 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:21.213093 (+ 3096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:21.213095 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:21.213096 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:21.213106 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:21.214963 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:21.215447 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:21.215448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:21.215449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:21.215510 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:21.215765 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:21.215766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:21.215766 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:21.215897 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:21.215980 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:21.216161 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 15:20:21.216235 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":107,"threads_started":1}
I20260925 15:20:21.218442 5644 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58228:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:20:21.218750 5644 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:20:21.220641 6061 tablet_service.cc:1467] Processing CreateTablet for tablet 89898d3e40b74d8da91f203a9703940c (DEFAULT_TABLE table=test-table [id=b3347fc6d213448b9bb9373de61348e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:20:21.220644 5921 tablet_service.cc:1467] Processing CreateTablet for tablet 89898d3e40b74d8da91f203a9703940c (DEFAULT_TABLE table=test-table [id=b3347fc6d213448b9bb9373de61348e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:20:21.220642 5780 tablet_service.cc:1467] Processing CreateTablet for tablet 89898d3e40b74d8da91f203a9703940c (DEFAULT_TABLE table=test-table [id=b3347fc6d213448b9bb9373de61348e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:20:21.220831 5921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89898d3e40b74d8da91f203a9703940c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:21.220827 5780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89898d3e40b74d8da91f203a9703940c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:21.220844 6061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89898d3e40b74d8da91f203a9703940c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:21.222119 6238 tablet_bootstrap.cc:492] T 89898d3e40b74d8da91f203a9703940c P 074cc39baf68454099b816ad22fddc74: Bootstrap starting.
I20260925 15:20:21.222353 6239 tablet_bootstrap.cc:492] T 89898d3e40b74d8da91f203a9703940c P ddb9ca84c5f740ba8983704a4381c165: Bootstrap starting.
I20260925 15:20:21.222724 6239 tablet_bootstrap.cc:654] T 89898d3e40b74d8da91f203a9703940c P ddb9ca84c5f740ba8983704a4381c165: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:21.222725 6238 tablet_bootstrap.cc:654] T 89898d3e40b74d8da91f203a9703940c P 074cc39baf68454099b816ad22fddc74: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:21.223047 6240 tablet_bootstrap.cc:492] T 89898d3e40b74d8da91f203a9703940c P 9ef0e719f724454ba531038d0612a66b: Bootstrap starting.
I20260925 15:20:21.223378 6240 tablet_bootstrap.cc:654] T 89898d3e40b74d8da91f203a9703940c P 9ef0e719f724454ba531038d0612a66b: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:21.223472 6238 tablet_bootstrap.cc:492] T 89898d3e40b74d8da91f203a9703940c P 074cc39baf68454099b816ad22fddc74: No bootstrap required, opened a new log
I20260925 15:20:21.223541 6238 ts_tablet_manager.cc:1397] T 89898d3e40b74d8da91f203a9703940c P 074cc39baf68454099b816ad22fddc74: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:20:21.223646 6239 tablet_bootstrap.cc:492] T 89898d3e40b74d8da91f203a9703940c P ddb9ca84c5f740ba8983704a4381c165: No bootstrap required, opened a new log
I20260925 15:20:21.223706 6239 ts_tablet_manager.cc:1397] T 89898d3e40b74d8da91f203a9703940c P ddb9ca84c5f740ba8983704a4381c165: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:21.223683 6238 raft_consensus.cc:348] T 89898d3e40b74d8da91f203a9703940c P 074cc39baf68454099b816ad22fddc74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } } peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } }
I20260925 15:20:21.223778 6238 raft_consensus.cc:374] T 89898d3e40b74d8da91f203a9703940c P 074cc39baf68454099b816ad22fddc74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:21.223811 6238 raft_consensus.cc:729] T 89898d3e40b74d8da91f203a9703940c P 074cc39baf68454099b816ad22fddc74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074cc39baf68454099b816ad22fddc74, State: Initialized, Role: FOLLOWER
I20260925 15:20:21.223868 6238 consensus_queue.cc:260] T 89898d3e40b74d8da91f203a9703940c P 074cc39baf68454099b816ad22fddc74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } } peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } }
I20260925 15:20:21.223856 6239 raft_consensus.cc:348] T 89898d3e40b74d8da91f203a9703940c P ddb9ca84c5f740ba8983704a4381c165 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } } peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } }
I20260925 15:20:21.223933 6239 raft_consensus.cc:374] T 89898d3e40b74d8da91f203a9703940c P ddb9ca84c5f740ba8983704a4381c165 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:21.223956 6239 raft_consensus.cc:729] T 89898d3e40b74d8da91f203a9703940c P ddb9ca84c5f740ba8983704a4381c165 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddb9ca84c5f740ba8983704a4381c165, State: Initialized, Role: FOLLOWER
I20260925 15:20:21.224030 6239 consensus_queue.cc:260] T 89898d3e40b74d8da91f203a9703940c P ddb9ca84c5f740ba8983704a4381c165 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } } peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } }
I20260925 15:20:21.224085 6238 ts_tablet_manager.cc:1428] T 89898d3e40b74d8da91f203a9703940c P 074cc39baf68454099b816ad22fddc74: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 15:20:21.224372 6239 ts_tablet_manager.cc:1428] T 89898d3e40b74d8da91f203a9703940c P ddb9ca84c5f740ba8983704a4381c165: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:21.224862 6240 tablet_bootstrap.cc:492] T 89898d3e40b74d8da91f203a9703940c P 9ef0e719f724454ba531038d0612a66b: No bootstrap required, opened a new log
I20260925 15:20:21.224925 6240 ts_tablet_manager.cc:1397] T 89898d3e40b74d8da91f203a9703940c P 9ef0e719f724454ba531038d0612a66b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:20:21.225052 6240 raft_consensus.cc:348] T 89898d3e40b74d8da91f203a9703940c P 9ef0e719f724454ba531038d0612a66b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } } peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } }
I20260925 15:20:21.225132 6240 raft_consensus.cc:374] T 89898d3e40b74d8da91f203a9703940c P 9ef0e719f724454ba531038d0612a66b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:21.225152 6240 raft_consensus.cc:729] T 89898d3e40b74d8da91f203a9703940c P 9ef0e719f724454ba531038d0612a66b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef0e719f724454ba531038d0612a66b, State: Initialized, Role: FOLLOWER
I20260925 15:20:21.225221 6240 consensus_queue.cc:260] T 89898d3e40b74d8da91f203a9703940c P 9ef0e719f724454ba531038d0612a66b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } } peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } }
I20260925 15:20:21.225322 6240 ts_tablet_manager.cc:1428] T 89898d3e40b74d8da91f203a9703940c P 9ef0e719f724454ba531038d0612a66b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:21.416067 6215 raft_consensus.cc:482] T 89898d3e40b74d8da91f203a9703940c P 9ef0e719f724454ba531038d0612a66b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:20:21.416275 6215 raft_consensus.cc:504] T 89898d3e40b74d8da91f203a9703940c P 9ef0e719f724454ba531038d0612a66b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } } peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } }
I20260925 15:20:21.416586 6215 leader_election.cc:288] T 89898d3e40b74d8da91f203a9703940c P 9ef0e719f724454ba531038d0612a66b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddb9ca84c5f740ba8983704a4381c165 (127.3.211.194:35019), 074cc39baf68454099b816ad22fddc74 (127.3.211.195:38695)
I20260925 15:20:21.416966 5940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89898d3e40b74d8da91f203a9703940c" candidate_uuid: "9ef0e719f724454ba531038d0612a66b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddb9ca84c5f740ba8983704a4381c165" is_pre_election: true
I20260925 15:20:21.416967 6080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89898d3e40b74d8da91f203a9703940c" candidate_uuid: "9ef0e719f724454ba531038d0612a66b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074cc39baf68454099b816ad22fddc74" is_pre_election: true
I20260925 15:20:21.417083 5940 raft_consensus.cc:2454] T 89898d3e40b74d8da91f203a9703940c P ddb9ca84c5f740ba8983704a4381c165 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ef0e719f724454ba531038d0612a66b in term 0.
I20260925 15:20:21.417109 6080 raft_consensus.cc:2454] T 89898d3e40b74d8da91f203a9703940c P 074cc39baf68454099b816ad22fddc74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ef0e719f724454ba531038d0612a66b in term 0.
I20260925 15:20:21.417339 5733 leader_election.cc:302] T 89898d3e40b74d8da91f203a9703940c P 9ef0e719f724454ba531038d0612a66b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ef0e719f724454ba531038d0612a66b, ddb9ca84c5f740ba8983704a4381c165; no voters:
I20260925 15:20:21.417474 6215 raft_consensus.cc:2781] T 89898d3e40b74d8da91f203a9703940c P 9ef0e719f724454ba531038d0612a66b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:20:21.417541 6215 raft_consensus.cc:482] T 89898d3e40b74d8da91f203a9703940c P 9ef0e719f724454ba531038d0612a66b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:20:21.417570 6215 raft_consensus.cc:3037] T 89898d3e40b74d8da91f203a9703940c P 9ef0e719f724454ba531038d0612a66b [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:21.418370 6215 raft_consensus.cc:504] T 89898d3e40b74d8da91f203a9703940c P 9ef0e719f724454ba531038d0612a66b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } } peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } }
I20260925 15:20:21.418550 6215 leader_election.cc:288] T 89898d3e40b74d8da91f203a9703940c P 9ef0e719f724454ba531038d0612a66b [CANDIDATE]: Term 1 election: Requested vote from peers ddb9ca84c5f740ba8983704a4381c165 (127.3.211.194:35019), 074cc39baf68454099b816ad22fddc74 (127.3.211.195:38695)
I20260925 15:20:21.418718 5940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89898d3e40b74d8da91f203a9703940c" candidate_uuid: "9ef0e719f724454ba531038d0612a66b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddb9ca84c5f740ba8983704a4381c165"
I20260925 15:20:21.418845 5940 raft_consensus.cc:3037] T 89898d3e40b74d8da91f203a9703940c P ddb9ca84c5f740ba8983704a4381c165 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:21.418839 6080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89898d3e40b74d8da91f203a9703940c" candidate_uuid: "9ef0e719f724454ba531038d0612a66b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074cc39baf68454099b816ad22fddc74"
I20260925 15:20:21.418910 6080 raft_consensus.cc:3037] T 89898d3e40b74d8da91f203a9703940c P 074cc39baf68454099b816ad22fddc74 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:21.419871 6080 raft_consensus.cc:2454] T 89898d3e40b74d8da91f203a9703940c P 074cc39baf68454099b816ad22fddc74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ef0e719f724454ba531038d0612a66b in term 1.
I20260925 15:20:21.420130 5735 leader_election.cc:302] T 89898d3e40b74d8da91f203a9703940c P 9ef0e719f724454ba531038d0612a66b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074cc39baf68454099b816ad22fddc74, 9ef0e719f724454ba531038d0612a66b; no voters:
I20260925 15:20:21.420197 5940 raft_consensus.cc:2454] T 89898d3e40b74d8da91f203a9703940c P ddb9ca84c5f740ba8983704a4381c165 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ef0e719f724454ba531038d0612a66b in term 1.
I20260925 15:20:21.420265 6215 raft_consensus.cc:2781] T 89898d3e40b74d8da91f203a9703940c P 9ef0e719f724454ba531038d0612a66b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:20:21.420320 6215 raft_consensus.cc:686] T 89898d3e40b74d8da91f203a9703940c P 9ef0e719f724454ba531038d0612a66b [term 1 LEADER]: Becoming Leader. State: Replica: 9ef0e719f724454ba531038d0612a66b, State: Running, Role: LEADER
I20260925 15:20:21.420408 6215 consensus_queue.cc:237] T 89898d3e40b74d8da91f203a9703940c P 9ef0e719f724454ba531038d0612a66b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } } peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } }
I20260925 15:20:21.421123 5644 catalog_manager.cc:5495] T 89898d3e40b74d8da91f203a9703940c P 9ef0e719f724454ba531038d0612a66b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ef0e719f724454ba531038d0612a66b (127.3.211.193). New cstate: current_term: 1 leader_uuid: "9ef0e719f724454ba531038d0612a66b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } health_report { overall_health: UNKNOWN } } }
I20260925 15:20:21.458797 6248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:21.455941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52404 (local address 127.3.211.193:34301)
0925 15:20:21.456105 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:21.456108 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:21.456116 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:20:21.456147 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:21.456147 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:21.456157 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:21.456221 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:20:21.456511 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:21.457013 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:21.457976 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:21.458176 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:21.458240 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:21.458451 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:20:21.458480 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:20:21.458521 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:21.458634 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 15:20:21.458654 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":111,"threads_started":1}
I20260925 15:20:21.461450 6080 raft_consensus.cc:1261] T 89898d3e40b74d8da91f203a9703940c P 074cc39baf68454099b816ad22fddc74 [term 1 FOLLOWER]: Refusing update from remote peer 9ef0e719f724454ba531038d0612a66b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:20:21.461463 5940 raft_consensus.cc:1261] T 89898d3e40b74d8da91f203a9703940c P ddb9ca84c5f740ba8983704a4381c165 [term 1 FOLLOWER]: Refusing update from remote peer 9ef0e719f724454ba531038d0612a66b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:20:21.461879 6215 consensus_queue.cc:1035] T 89898d3e40b74d8da91f203a9703940c P 9ef0e719f724454ba531038d0612a66b [LEADER]: Connected to new peer: Peer: permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:20:21.462116 6210 consensus_queue.cc:1035] T 89898d3e40b74d8da91f203a9703940c P 9ef0e719f724454ba531038d0612a66b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:20:21.473400 5643 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:20:21.473522 5643 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:20:21.474153 5643 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bba60aadb5c245e6bb6d4680cdfa0adc: Sending DeleteTablet for 3 replicas of tablet 89898d3e40b74d8da91f203a9703940c
I20260925 15:20:21.474412 5780 tablet_service.cc:1514] Processing DeleteTablet for tablet 89898d3e40b74d8da91f203a9703940c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:20:21 UTC) from {username='kudu', principal='kudu/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:52376
I20260925 15:20:21.474447 6061 tablet_service.cc:1514] Processing DeleteTablet for tablet 89898d3e40b74d8da91f203a9703940c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:20:21 UTC) from {username='kudu', principal='kudu/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:59108
I20260925 15:20:21.474542 6227 tablet_replica.cc:330] T 89898d3e40b74d8da91f203a9703940c P 9ef0e719f724454ba531038d0612a66b: stopping tablet replica
I20260925 15:20:21.474597 6227 raft_consensus.cc:2229] T 89898d3e40b74d8da91f203a9703940c P 9ef0e719f724454ba531038d0612a66b [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:20:21.474611 6228 tablet_replica.cc:330] T 89898d3e40b74d8da91f203a9703940c P 074cc39baf68454099b816ad22fddc74: stopping tablet replica
I20260925 15:20:21.474653 6227 raft_consensus.cc:2258] T 89898d3e40b74d8da91f203a9703940c P 9ef0e719f724454ba531038d0612a66b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:20:21.474670 6228 raft_consensus.cc:2229] T 89898d3e40b74d8da91f203a9703940c P 074cc39baf68454099b816ad22fddc74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:20:21.474732 6228 raft_consensus.cc:2258] T 89898d3e40b74d8da91f203a9703940c P 074cc39baf68454099b816ad22fddc74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:20:21.474967 6227 ts_tablet_manager.cc:1905] T 89898d3e40b74d8da91f203a9703940c P 9ef0e719f724454ba531038d0612a66b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:20:21.474937 5643 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58228:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:20:21.475070 5643 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:20:21.475320 5921 tablet_service.cc:1514] Processing DeleteTablet for tablet 89898d3e40b74d8da91f203a9703940c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:20:21 UTC) from {username='kudu', principal='kudu/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:47456
I20260925 15:20:21.475425 6229 tablet_replica.cc:330] T 89898d3e40b74d8da91f203a9703940c P ddb9ca84c5f740ba8983704a4381c165: stopping tablet replica
I20260925 15:20:21.475466 6228 ts_tablet_manager.cc:1905] T 89898d3e40b74d8da91f203a9703940c P 074cc39baf68454099b816ad22fddc74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:20:21.475477 6229 raft_consensus.cc:2229] T 89898d3e40b74d8da91f203a9703940c P ddb9ca84c5f740ba8983704a4381c165 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:20:21.475530 6229 raft_consensus.cc:2258] T 89898d3e40b74d8da91f203a9703940c P ddb9ca84c5f740ba8983704a4381c165 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:20:21.476536 6061 tablet_service.cc:1467] Processing CreateTablet for tablet e47de1759b6240499e6fa5b7db4955e4 (DEFAULT_TABLE table=test-table [id=f9506d12fdac414ca341dab0820f8b8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:20:21.476644 6229 ts_tablet_manager.cc:1905] T 89898d3e40b74d8da91f203a9703940c P ddb9ca84c5f740ba8983704a4381c165: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:20:21.476655 6061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e47de1759b6240499e6fa5b7db4955e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:21.476660 6227 ts_tablet_manager.cc:1918] T 89898d3e40b74d8da91f203a9703940c P 9ef0e719f724454ba531038d0612a66b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:20:21.476708 6227 log.cc:1196] T 89898d3e40b74d8da91f203a9703940c P 9ef0e719f724454ba531038d0612a66b: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-0/wal/wals/89898d3e40b74d8da91f203a9703940c
I20260925 15:20:21.476938 6227 ts_tablet_manager.cc:1939] T 89898d3e40b74d8da91f203a9703940c P 9ef0e719f724454ba531038d0612a66b: Deleting consensus metadata
I20260925 15:20:21.477196 6228 ts_tablet_manager.cc:1918] T 89898d3e40b74d8da91f203a9703940c P 074cc39baf68454099b816ad22fddc74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:20:21.477245 6228 log.cc:1196] T 89898d3e40b74d8da91f203a9703940c P 074cc39baf68454099b816ad22fddc74: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-2/wal/wals/89898d3e40b74d8da91f203a9703940c
I20260925 15:20:21.477345 5628 catalog_manager.cc:4841] TS 9ef0e719f724454ba531038d0612a66b (127.3.211.193:34301): tablet 89898d3e40b74d8da91f203a9703940c (table test-table [id=b3347fc6d213448b9bb9373de61348e1]) successfully deleted
I20260925 15:20:21.477468 6228 ts_tablet_manager.cc:1939] T 89898d3e40b74d8da91f203a9703940c P 074cc39baf68454099b816ad22fddc74: Deleting consensus metadata
I20260925 15:20:21.477790 5631 catalog_manager.cc:4841] TS 074cc39baf68454099b816ad22fddc74 (127.3.211.195:38695): tablet 89898d3e40b74d8da91f203a9703940c (table test-table [id=b3347fc6d213448b9bb9373de61348e1]) successfully deleted
I20260925 15:20:21.477775 5780 tablet_service.cc:1467] Processing CreateTablet for tablet e47de1759b6240499e6fa5b7db4955e4 (DEFAULT_TABLE table=test-table [id=f9506d12fdac414ca341dab0820f8b8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:20:21.477878 5780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e47de1759b6240499e6fa5b7db4955e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:21.477921 6238 tablet_bootstrap.cc:492] T e47de1759b6240499e6fa5b7db4955e4 P 074cc39baf68454099b816ad22fddc74: Bootstrap starting.
I20260925 15:20:21.478152 6229 ts_tablet_manager.cc:1918] T 89898d3e40b74d8da91f203a9703940c P ddb9ca84c5f740ba8983704a4381c165: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:20:21.478197 6238 tablet_bootstrap.cc:654] T e47de1759b6240499e6fa5b7db4955e4 P 074cc39baf68454099b816ad22fddc74: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:21.478217 6229 log.cc:1196] T 89898d3e40b74d8da91f203a9703940c P ddb9ca84c5f740ba8983704a4381c165: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-1/wal/wals/89898d3e40b74d8da91f203a9703940c
I20260925 15:20:21.478431 6229 ts_tablet_manager.cc:1939] T 89898d3e40b74d8da91f203a9703940c P ddb9ca84c5f740ba8983704a4381c165: Deleting consensus metadata
I20260925 15:20:21.478797 5629 catalog_manager.cc:4841] TS ddb9ca84c5f740ba8983704a4381c165 (127.3.211.194:35019): tablet 89898d3e40b74d8da91f203a9703940c (table test-table [id=b3347fc6d213448b9bb9373de61348e1]) successfully deleted
I20260925 15:20:21.478869 6238 tablet_bootstrap.cc:492] T e47de1759b6240499e6fa5b7db4955e4 P 074cc39baf68454099b816ad22fddc74: No bootstrap required, opened a new log
I20260925 15:20:21.478864 5921 tablet_service.cc:1467] Processing CreateTablet for tablet e47de1759b6240499e6fa5b7db4955e4 (DEFAULT_TABLE table=test-table [id=f9506d12fdac414ca341dab0820f8b8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:20:21.478910 6238 ts_tablet_manager.cc:1397] T e47de1759b6240499e6fa5b7db4955e4 P 074cc39baf68454099b816ad22fddc74: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:21.478955 5921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e47de1759b6240499e6fa5b7db4955e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:21.479020 6238 raft_consensus.cc:348] T e47de1759b6240499e6fa5b7db4955e4 P 074cc39baf68454099b816ad22fddc74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } } peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } }
I20260925 15:20:21.479102 6238 raft_consensus.cc:374] T e47de1759b6240499e6fa5b7db4955e4 P 074cc39baf68454099b816ad22fddc74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:21.479128 6238 raft_consensus.cc:729] T e47de1759b6240499e6fa5b7db4955e4 P 074cc39baf68454099b816ad22fddc74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074cc39baf68454099b816ad22fddc74, State: Initialized, Role: FOLLOWER
I20260925 15:20:21.479179 6238 consensus_queue.cc:260] T e47de1759b6240499e6fa5b7db4955e4 P 074cc39baf68454099b816ad22fddc74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } } peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } }
I20260925 15:20:21.479302 6238 ts_tablet_manager.cc:1428] T e47de1759b6240499e6fa5b7db4955e4 P 074cc39baf68454099b816ad22fddc74: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 15:20:21.479336 6240 tablet_bootstrap.cc:492] T e47de1759b6240499e6fa5b7db4955e4 P 9ef0e719f724454ba531038d0612a66b: Bootstrap starting.
I20260925 15:20:21.479660 6240 tablet_bootstrap.cc:654] T e47de1759b6240499e6fa5b7db4955e4 P 9ef0e719f724454ba531038d0612a66b: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:21.480193 6239 tablet_bootstrap.cc:492] T e47de1759b6240499e6fa5b7db4955e4 P ddb9ca84c5f740ba8983704a4381c165: Bootstrap starting.
I20260925 15:20:21.480374 6240 tablet_bootstrap.cc:492] T e47de1759b6240499e6fa5b7db4955e4 P 9ef0e719f724454ba531038d0612a66b: No bootstrap required, opened a new log
I20260925 15:20:21.480434 6240 ts_tablet_manager.cc:1397] T e47de1759b6240499e6fa5b7db4955e4 P 9ef0e719f724454ba531038d0612a66b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:21.480551 6240 raft_consensus.cc:348] T e47de1759b6240499e6fa5b7db4955e4 P 9ef0e719f724454ba531038d0612a66b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } } peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } }
I20260925 15:20:21.480633 6239 tablet_bootstrap.cc:654] T e47de1759b6240499e6fa5b7db4955e4 P ddb9ca84c5f740ba8983704a4381c165: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:21.480641 6240 raft_consensus.cc:374] T e47de1759b6240499e6fa5b7db4955e4 P 9ef0e719f724454ba531038d0612a66b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:21.480667 6240 raft_consensus.cc:729] T e47de1759b6240499e6fa5b7db4955e4 P 9ef0e719f724454ba531038d0612a66b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef0e719f724454ba531038d0612a66b, State: Initialized, Role: FOLLOWER
I20260925 15:20:21.480724 6240 consensus_queue.cc:260] T e47de1759b6240499e6fa5b7db4955e4 P 9ef0e719f724454ba531038d0612a66b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } } peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } }
I20260925 15:20:21.480842 6240 ts_tablet_manager.cc:1428] T e47de1759b6240499e6fa5b7db4955e4 P 9ef0e719f724454ba531038d0612a66b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:21.481330 6239 tablet_bootstrap.cc:492] T e47de1759b6240499e6fa5b7db4955e4 P ddb9ca84c5f740ba8983704a4381c165: No bootstrap required, opened a new log
I20260925 15:20:21.481408 6239 ts_tablet_manager.cc:1397] T e47de1759b6240499e6fa5b7db4955e4 P ddb9ca84c5f740ba8983704a4381c165: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:21.481559 6239 raft_consensus.cc:348] T e47de1759b6240499e6fa5b7db4955e4 P ddb9ca84c5f740ba8983704a4381c165 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } } peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } }
I20260925 15:20:21.481631 6239 raft_consensus.cc:374] T e47de1759b6240499e6fa5b7db4955e4 P ddb9ca84c5f740ba8983704a4381c165 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:21.481652 6239 raft_consensus.cc:729] T e47de1759b6240499e6fa5b7db4955e4 P ddb9ca84c5f740ba8983704a4381c165 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddb9ca84c5f740ba8983704a4381c165, State: Initialized, Role: FOLLOWER
I20260925 15:20:21.481724 6239 consensus_queue.cc:260] T e47de1759b6240499e6fa5b7db4955e4 P ddb9ca84c5f740ba8983704a4381c165 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } } peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } }
I20260925 15:20:21.481825 6239 ts_tablet_manager.cc:1428] T e47de1759b6240499e6fa5b7db4955e4 P ddb9ca84c5f740ba8983704a4381c165: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:21.537338 6245 raft_consensus.cc:482] T e47de1759b6240499e6fa5b7db4955e4 P ddb9ca84c5f740ba8983704a4381c165 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:20:21.537453 6245 raft_consensus.cc:504] T e47de1759b6240499e6fa5b7db4955e4 P ddb9ca84c5f740ba8983704a4381c165 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } } peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } }
I20260925 15:20:21.538002 6245 leader_election.cc:288] T e47de1759b6240499e6fa5b7db4955e4 P ddb9ca84c5f740ba8983704a4381c165 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ef0e719f724454ba531038d0612a66b (127.3.211.193:34301), 074cc39baf68454099b816ad22fddc74 (127.3.211.195:38695)
I20260925 15:20:21.541170 6257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:21.538302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:44613 (local address 127.3.211.195:38695)
0925 15:20:21.538539 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:21.538542 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:21.538550 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:20:21.538622 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:21.538623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:21.538635 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:21.538699 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:20:21.538870 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:21.539356 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:21.540405 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:21.540791 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:21.540881 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:21.540891 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:21.541041 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 15:20:21.541084 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":62,"threads_started":1}
I20260925 15:20:21.541266 6248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:21.538496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:47741 (local address 127.3.211.193:34301)
0925 15:20:21.538541 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:21.538543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:21.538552 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:20:21.538587 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:21.538587 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:21.538598 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:21.538675 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:20:21.538857 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:21.539383 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:21.540448 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:21.540848 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:21.540951 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:21.540964 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:21.541149 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 15:20:21.541190 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 15:20:21.541299 6255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:21.538206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:34301 (local address 127.3.211.194:47741)
0925 15:20:21.538336 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:21.538348 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:21.538408 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:21.538690 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:21.538691 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:21.538694 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:20:21.538774 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:21.538776 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:21.539494 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:21.539495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:21.540295 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:21.540297 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:21.540971 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:21.540972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:21.541043 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:21.541046 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:20:21.541139 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 15:20:21.541185 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":68,"threads_started":1}
I20260925 15:20:21.541190 6256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:21.538220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.195:38695 (local address 127.3.211.194:44613)
0925 15:20:21.538429 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:21.538438 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:21.538484 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:21.538703 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:21.538704 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:21.538706 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:20:21.538790 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:21.538792 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:21.539457 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:21.539459 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:21.540269 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:21.540271 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:21.540878 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:21.540879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:21.540960 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:21.540964 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:20:21.541035 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 15:20:21.541084 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":4,"thread_start_us":96,"threads_started":1}
I20260925 15:20:21.541894 6080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47de1759b6240499e6fa5b7db4955e4" candidate_uuid: "ddb9ca84c5f740ba8983704a4381c165" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074cc39baf68454099b816ad22fddc74" is_pre_election: true
I20260925 15:20:21.542035 6080 raft_consensus.cc:2454] T e47de1759b6240499e6fa5b7db4955e4 P 074cc39baf68454099b816ad22fddc74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddb9ca84c5f740ba8983704a4381c165 in term 0.
I20260925 15:20:21.542042 5801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47de1759b6240499e6fa5b7db4955e4" candidate_uuid: "ddb9ca84c5f740ba8983704a4381c165" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef0e719f724454ba531038d0612a66b" is_pre_election: true
I20260925 15:20:21.542174 5801 raft_consensus.cc:2454] T e47de1759b6240499e6fa5b7db4955e4 P 9ef0e719f724454ba531038d0612a66b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddb9ca84c5f740ba8983704a4381c165 in term 0.
I20260925 15:20:21.542224 5875 leader_election.cc:302] T e47de1759b6240499e6fa5b7db4955e4 P ddb9ca84c5f740ba8983704a4381c165 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074cc39baf68454099b816ad22fddc74, ddb9ca84c5f740ba8983704a4381c165; no voters:
I20260925 15:20:21.542358 6245 raft_consensus.cc:2781] T e47de1759b6240499e6fa5b7db4955e4 P ddb9ca84c5f740ba8983704a4381c165 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:20:21.542390 6245 raft_consensus.cc:482] T e47de1759b6240499e6fa5b7db4955e4 P ddb9ca84c5f740ba8983704a4381c165 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:20:21.542409 6245 raft_consensus.cc:3037] T e47de1759b6240499e6fa5b7db4955e4 P ddb9ca84c5f740ba8983704a4381c165 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:21.543090 6245 raft_consensus.cc:504] T e47de1759b6240499e6fa5b7db4955e4 P ddb9ca84c5f740ba8983704a4381c165 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } } peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } }
I20260925 15:20:21.543241 6245 leader_election.cc:288] T e47de1759b6240499e6fa5b7db4955e4 P ddb9ca84c5f740ba8983704a4381c165 [CANDIDATE]: Term 1 election: Requested vote from peers 9ef0e719f724454ba531038d0612a66b (127.3.211.193:34301), 074cc39baf68454099b816ad22fddc74 (127.3.211.195:38695)
I20260925 15:20:21.543390 5801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47de1759b6240499e6fa5b7db4955e4" candidate_uuid: "ddb9ca84c5f740ba8983704a4381c165" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef0e719f724454ba531038d0612a66b"
I20260925 15:20:21.543406 6080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e47de1759b6240499e6fa5b7db4955e4" candidate_uuid: "ddb9ca84c5f740ba8983704a4381c165" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074cc39baf68454099b816ad22fddc74"
I20260925 15:20:21.543478 5801 raft_consensus.cc:3037] T e47de1759b6240499e6fa5b7db4955e4 P 9ef0e719f724454ba531038d0612a66b [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:21.543478 6080 raft_consensus.cc:3037] T e47de1759b6240499e6fa5b7db4955e4 P 074cc39baf68454099b816ad22fddc74 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:21.544234 5801 raft_consensus.cc:2454] T e47de1759b6240499e6fa5b7db4955e4 P 9ef0e719f724454ba531038d0612a66b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddb9ca84c5f740ba8983704a4381c165 in term 1.
I20260925 15:20:21.544230 6080 raft_consensus.cc:2454] T e47de1759b6240499e6fa5b7db4955e4 P 074cc39baf68454099b816ad22fddc74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddb9ca84c5f740ba8983704a4381c165 in term 1.
I20260925 15:20:21.544449 5875 leader_election.cc:302] T e47de1759b6240499e6fa5b7db4955e4 P ddb9ca84c5f740ba8983704a4381c165 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074cc39baf68454099b816ad22fddc74, ddb9ca84c5f740ba8983704a4381c165; no voters:
I20260925 15:20:21.544603 6245 raft_consensus.cc:2781] T e47de1759b6240499e6fa5b7db4955e4 P ddb9ca84c5f740ba8983704a4381c165 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:20:21.544785 6245 raft_consensus.cc:686] T e47de1759b6240499e6fa5b7db4955e4 P ddb9ca84c5f740ba8983704a4381c165 [term 1 LEADER]: Becoming Leader. State: Replica: ddb9ca84c5f740ba8983704a4381c165, State: Running, Role: LEADER
I20260925 15:20:21.544875 6245 consensus_queue.cc:237] T e47de1759b6240499e6fa5b7db4955e4 P ddb9ca84c5f740ba8983704a4381c165 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } } peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } }
I20260925 15:20:21.545722 5643 catalog_manager.cc:5495] T e47de1759b6240499e6fa5b7db4955e4 P ddb9ca84c5f740ba8983704a4381c165 reported cstate change: term changed from 0 to 1, leader changed from <none> to ddb9ca84c5f740ba8983704a4381c165 (127.3.211.194). New cstate: current_term: 1 leader_uuid: "ddb9ca84c5f740ba8983704a4381c165" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddb9ca84c5f740ba8983704a4381c165" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 35019 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 } health_report { overall_health: UNKNOWN } } }
I20260925 15:20:21.560837 6260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:21.558040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47470 (local address 127.3.211.194:35019)
0925 15:20:21.558250 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:21.558251 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:21.558257 (+ 6us) server_negotiation.cc:407] Connection header received
0925 15:20:21.558291 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:21.558291 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:21.558310 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:21.558403 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:20:21.558507 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:21.559132 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:21.559879 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:21.560115 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:21.560194 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:21.560368 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:20:21.560486 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:20:21.560578 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:21.560620 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 15:20:21.560711 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":128,"threads_started":1}
I20260925 15:20:21.566756 6248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:21.563720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:50009 (local address 127.3.211.193:34301)
0925 15:20:21.563801 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:21.563803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:21.563971 (+ 168us) server_negotiation.cc:407] Connection header received
0925 15:20:21.564003 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:21.564004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:21.564014 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:21.564079 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:20:21.564272 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:21.564855 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:21.566026 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:21.566240 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:21.566313 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:21.566507 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:20:21.566536 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:20:21.566596 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:21.566667 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 15:20:21.566689 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 15:20:21.566785 6263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:21.563658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:34301 (local address 127.3.211.194:50009)
0925 15:20:21.563871 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:21.563881 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:21.563936 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:21.564086 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:21.564087 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:21.564088 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:20:21.564213 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:21.564215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:21.564939 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:21.564940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:21.565906 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:21.565909 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:21.566340 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:21.566341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:21.566401 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:21.566417 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:20:21.566616 (+ 199us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:20:21.566618 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 15:20:21.566665 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 15:20:21.566710 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":124,"threads_started":1}
I20260925 15:20:21.569182 6080 raft_consensus.cc:1261] T e47de1759b6240499e6fa5b7db4955e4 P 074cc39baf68454099b816ad22fddc74 [term 1 FOLLOWER]: Refusing update from remote peer ddb9ca84c5f740ba8983704a4381c165: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:20:21.569271 5801 raft_consensus.cc:1261] T e47de1759b6240499e6fa5b7db4955e4 P 9ef0e719f724454ba531038d0612a66b [term 1 FOLLOWER]: Refusing update from remote peer ddb9ca84c5f740ba8983704a4381c165: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:20:21.569509 6258 consensus_queue.cc:1035] T e47de1759b6240499e6fa5b7db4955e4 P ddb9ca84c5f740ba8983704a4381c165 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074cc39baf68454099b816ad22fddc74" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:20:21.569653 6245 consensus_queue.cc:1035] T e47de1759b6240499e6fa5b7db4955e4 P ddb9ca84c5f740ba8983704a4381c165 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ef0e719f724454ba531038d0612a66b" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:20:21.578006 6260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:21.575168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:51477 (local address 127.3.211.194:35019)
0925 15:20:21.575231 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:21.575233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:21.575429 (+ 196us) server_negotiation.cc:407] Connection header received
0925 15:20:21.575498 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:21.575499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:21.575512 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:21.575564 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:20:21.575763 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:21.576242 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:21.577250 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:21.577479 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:21.577573 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:21.577730 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:20:21.577767 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:20:21.577838 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:21.577902 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 15:20:21.577923 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 15:20:21.578024 6268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:21.575103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.194:35019 (local address 127.3.211.193:51477)
0925 15:20:21.575350 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:21.575360 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:21.575437 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:21.575616 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:21.575617 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:21.575619 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:20:21.575684 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:21.575686 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:21.576406 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:21.576407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:21.577144 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:21.577146 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:21.577578 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:21.577579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:21.577635 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:21.577641 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:20:21.577841 (+ 200us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:20:21.577843 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 15:20:21.577897 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 15:20:21.577945 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":100,"threads_started":1}
W20260925 15:20:21.578405 6212 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 15:20:23.100962 5642 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:20:23.101082 5642 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:20:23.102061 5642 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bba60aadb5c245e6bb6d4680cdfa0adc: Sending DeleteTablet for 3 replicas of tablet e47de1759b6240499e6fa5b7db4955e4
I20260925 15:20:23.102421 5780 tablet_service.cc:1514] Processing DeleteTablet for tablet e47de1759b6240499e6fa5b7db4955e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:20:23 UTC) from {username='kudu', principal='kudu/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:52376
I20260925 15:20:23.102406 5921 tablet_service.cc:1514] Processing DeleteTablet for tablet e47de1759b6240499e6fa5b7db4955e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:20:23 UTC) from {username='kudu', principal='kudu/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:47456
I20260925 15:20:23.102483 6061 tablet_service.cc:1514] Processing DeleteTablet for tablet e47de1759b6240499e6fa5b7db4955e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:20:23 UTC) from {username='kudu', principal='kudu/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:59108
I20260925 15:20:23.102643 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 5716
I20260925 15:20:23.103525 6292 tablet_replica.cc:330] T e47de1759b6240499e6fa5b7db4955e4 P ddb9ca84c5f740ba8983704a4381c165: stopping tablet replica
I20260925 15:20:23.103592 6292 raft_consensus.cc:2229] T e47de1759b6240499e6fa5b7db4955e4 P ddb9ca84c5f740ba8983704a4381c165 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:20:23.103698 6292 raft_consensus.cc:2258] T e47de1759b6240499e6fa5b7db4955e4 P ddb9ca84c5f740ba8983704a4381c165 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:20:23.104009 6292 ts_tablet_manager.cc:1905] T e47de1759b6240499e6fa5b7db4955e4 P ddb9ca84c5f740ba8983704a4381c165: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:20:23.106345 6292 ts_tablet_manager.cc:1918] T e47de1759b6240499e6fa5b7db4955e4 P ddb9ca84c5f740ba8983704a4381c165: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 15:20:23.106427 6292 log.cc:1196] T e47de1759b6240499e6fa5b7db4955e4 P ddb9ca84c5f740ba8983704a4381c165: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-1/wal/wals/e47de1759b6240499e6fa5b7db4955e4
I20260925 15:20:23.106683 6292 ts_tablet_manager.cc:1939] T e47de1759b6240499e6fa5b7db4955e4 P ddb9ca84c5f740ba8983704a4381c165: Deleting consensus metadata
I20260925 15:20:23.107192 5629 catalog_manager.cc:4841] TS ddb9ca84c5f740ba8983704a4381c165 (127.3.211.194:35019): tablet e47de1759b6240499e6fa5b7db4955e4 (table test-table [id=f9506d12fdac414ca341dab0820f8b8d]) successfully deleted
I20260925 15:20:23.108506 6294 tablet_replica.cc:330] T e47de1759b6240499e6fa5b7db4955e4 P 074cc39baf68454099b816ad22fddc74: stopping tablet replica
I20260925 15:20:23.108577 6294 raft_consensus.cc:2229] T e47de1759b6240499e6fa5b7db4955e4 P 074cc39baf68454099b816ad22fddc74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:20:23.108642 6294 raft_consensus.cc:2258] T e47de1759b6240499e6fa5b7db4955e4 P 074cc39baf68454099b816ad22fddc74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:20:23.108909 6294 ts_tablet_manager.cc:1905] T e47de1759b6240499e6fa5b7db4955e4 P 074cc39baf68454099b816ad22fddc74: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 15:20:23.110152 5628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.211.193:34301 (error 108)
I20260925 15:20:23.110431 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 5856
I20260925 15:20:23.110564 6294 ts_tablet_manager.cc:1918] T e47de1759b6240499e6fa5b7db4955e4 P 074cc39baf68454099b816ad22fddc74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 15:20:23.110626 6294 log.cc:1196] T e47de1759b6240499e6fa5b7db4955e4 P 074cc39baf68454099b816ad22fddc74: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790349610595785-3919-0/minicluster-data/ts-2/wal/wals/e47de1759b6240499e6fa5b7db4955e4
I20260925 15:20:23.110843 6294 ts_tablet_manager.cc:1939] T e47de1759b6240499e6fa5b7db4955e4 P 074cc39baf68454099b816ad22fddc74: Deleting consensus metadata
I20260925 15:20:23.111299 5631 catalog_manager.cc:4841] TS 074cc39baf68454099b816ad22fddc74 (127.3.211.195:38695): tablet e47de1759b6240499e6fa5b7db4955e4 (table test-table [id=f9506d12fdac414ca341dab0820f8b8d]) successfully deleted
I20260925 15:20:23.114789 6296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:23.114515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:34301 (local address 127.0.0.1:57554)
0925 15:20:23.114664 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:23.114728 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.193:34301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":84,"threads_started":1}
W20260925 15:20:23.114961 5628 catalog_manager.cc:4584] TS 9ef0e719f724454ba531038d0612a66b (127.3.211.193:34301): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e47de1759b6240499e6fa5b7db4955e4: Network error: Client connection negotiation failed: client connection to 127.3.211.193:34301: connect: Connection refused (error 111)
I20260925 15:20:23.117197 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 5996
I20260925 15:20:23.123308 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 5612
2026-09-25T15:20:23Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6111 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:20:23 dist-test-slave-4bk3 krb5kdc[6299](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:20:23 dist-test-slave-4bk3 krb5kdc[6299](info): set up 2 sockets
Sep 25 15:20:23 dist-test-slave-4bk3 krb5kdc[6299](info): commencing operation
krb5kdc: starting...
W20260925 15:20:24.209082 3919 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Sep 25 15:20:24 dist-test-slave-4bk3 krb5kdc[6299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:20:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:20:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.211.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
I20260925 15:20:24.334975 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:35285
--webserver_interface=127.3.211.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:36281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.211.254:35285
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:20:24.420912 6312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:24.421151 6312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:24.421178 6312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:24.422519 6312 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:20:24.422585 6312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:24.422600 6312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:24.422613 6312 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:20:24.422632 6312 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:20:24.424384 6312 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:36281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.211.254:35285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:35285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.211.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:24.424693 6312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:24.424986 6312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:24.430423 6317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:24.430423 6320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:24.430481 6318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:24.430898 6312 server_base.cc:1034] running on GCE node
I20260925 15:20:24.431152 6312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:24.431443 6312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:24.432578 6312 hybrid_clock.cc:648] HybridClock initialized: now 1790349624432566 us; error 36 us; skew 500 ppm
Sep 25 15:20:24 dist-test-slave-4bk3 krb5kdc[6299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349624, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:24.435238 6312 init.cc:377] Logged in from keytab as kudu/127.3.211.254@KRBTEST.COM (short username kudu)
I20260925 15:20:24.435694 6312 webserver.cc:460] Webserver started at http://127.3.211.254:41273/ using document root <none> and password file <none>
I20260925 15:20:24.435883 6312 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:24.435925 6312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:24.436018 6312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:24.437173 6312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/master-0/data/instance:
uuid: "17d8497cfa044ad582f23ad7a6d7129e"
format_stamp: "Formatted at 2026-09-25 15:20:24 on dist-test-slave-4bk3"
server_key: "32ddcb11a07d9e71fcf01beeb2adb632"
server_key_iv: "6b73ee1b792077c210252bb447b10236"
server_key_version: "encryptionkey@0"
I20260925 15:20:24.437605 6312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/master-0/wal/instance:
uuid: "17d8497cfa044ad582f23ad7a6d7129e"
format_stamp: "Formatted at 2026-09-25 15:20:24 on dist-test-slave-4bk3"
server_key: "32ddcb11a07d9e71fcf01beeb2adb632"
server_key_iv: "6b73ee1b792077c210252bb447b10236"
server_key_version: "encryptionkey@0"
I20260925 15:20:24.438999 6312 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:20:24.439831 6327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:24.440152 6312 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:24.440236 6312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/master-0/wal
uuid: "17d8497cfa044ad582f23ad7a6d7129e"
format_stamp: "Formatted at 2026-09-25 15:20:24 on dist-test-slave-4bk3"
server_key: "32ddcb11a07d9e71fcf01beeb2adb632"
server_key_iv: "6b73ee1b792077c210252bb447b10236"
server_key_version: "encryptionkey@0"
I20260925 15:20:24.440302 6312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:24.452416 6312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:24.454872 6312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:24.455003 6312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:24.459277 6312 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.254:35285
I20260925 15:20:24.459312 6379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.254:35285 every 8 connection(s)
I20260925 15:20:24.459707 6312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
I20260925 15:20:24.460564 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 6312
I20260925 15:20:24.460662 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/master-0/wal/instance
I20260925 15:20:24.460857 3919 external_mini_cluster.cc:1428] Setting key 18f7e13b8a57b45bd6da31c498879c18
I20260925 15:20:24.460852 6380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:24.464368 6380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17d8497cfa044ad582f23ad7a6d7129e: Bootstrap starting.
I20260925 15:20:24.465039 6380 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17d8497cfa044ad582f23ad7a6d7129e: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:24.465390 6380 log.cc:824] T 00000000000000000000000000000000 P 17d8497cfa044ad582f23ad7a6d7129e: Log is configured to *not* fsync() on all Append() calls
I20260925 15:20:24.466365 6380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17d8497cfa044ad582f23ad7a6d7129e: No bootstrap required, opened a new log
Sep 25 15:20:24 dist-test-slave-4bk3 krb5kdc[6299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:20:24.469298 6380 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17d8497cfa044ad582f23ad7a6d7129e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17d8497cfa044ad582f23ad7a6d7129e" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 35285 } }
I20260925 15:20:24.469512 6380 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17d8497cfa044ad582f23ad7a6d7129e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:24.469573 6380 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17d8497cfa044ad582f23ad7a6d7129e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17d8497cfa044ad582f23ad7a6d7129e, State: Initialized, Role: FOLLOWER
I20260925 15:20:24.469759 6380 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17d8497cfa044ad582f23ad7a6d7129e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17d8497cfa044ad582f23ad7a6d7129e" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 35285 } }
I20260925 15:20:24.469835 6380 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17d8497cfa044ad582f23ad7a6d7129e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:20:24.469905 6380 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17d8497cfa044ad582f23ad7a6d7129e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:20:24.469964 6380 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17d8497cfa044ad582f23ad7a6d7129e [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:24.470911 6380 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17d8497cfa044ad582f23ad7a6d7129e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17d8497cfa044ad582f23ad7a6d7129e" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 35285 } }
I20260925 15:20:24.471045 6380 leader_election.cc:302] T 00000000000000000000000000000000 P 17d8497cfa044ad582f23ad7a6d7129e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17d8497cfa044ad582f23ad7a6d7129e; no voters:
I20260925 15:20:24.471261 6380 leader_election.cc:288] T 00000000000000000000000000000000 P 17d8497cfa044ad582f23ad7a6d7129e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:20:24.471354 6385 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17d8497cfa044ad582f23ad7a6d7129e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:20:24.471580 6380 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17d8497cfa044ad582f23ad7a6d7129e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:20:24.471580 6385 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17d8497cfa044ad582f23ad7a6d7129e [term 1 LEADER]: Becoming Leader. State: Replica: 17d8497cfa044ad582f23ad7a6d7129e, State: Running, Role: LEADER
I20260925 15:20:24.471817 6385 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17d8497cfa044ad582f23ad7a6d7129e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17d8497cfa044ad582f23ad7a6d7129e" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 35285 } }
I20260925 15:20:24.472187 6386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17d8497cfa044ad582f23ad7a6d7129e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17d8497cfa044ad582f23ad7a6d7129e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17d8497cfa044ad582f23ad7a6d7129e" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 35285 } } }
I20260925 15:20:24.472275 6386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17d8497cfa044ad582f23ad7a6d7129e [sys.catalog]: This master's current role is: LEADER
I20260925 15:20:24.472465 6387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17d8497cfa044ad582f23ad7a6d7129e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17d8497cfa044ad582f23ad7a6d7129e. Latest consensus state: current_term: 1 leader_uuid: "17d8497cfa044ad582f23ad7a6d7129e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17d8497cfa044ad582f23ad7a6d7129e" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 35285 } } }
I20260925 15:20:24.472536 6387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17d8497cfa044ad582f23ad7a6d7129e [sys.catalog]: This master's current role is: LEADER
I20260925 15:20:24.473080 6393 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:20:24.473763 6393 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:20:24.474102 6383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:24.461457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52420 (local address 127.3.211.254:35285)
0925 15:20:24.461657 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:24.461667 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:24.461701 (+ 34us) server_negotiation.cc:407] Connection header received
0925 15:20:24.462163 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:24.462173 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:24.462241 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:24.462405 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:24.463078 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:24.464477 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:24.465320 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:24.466006 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:24.466111 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:24.469245 (+ 3134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:24.469250 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:24.469250 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:24.469264 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:24.471588 (+ 2324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:24.472155 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:24.472158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:24.472158 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:24.472291 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:24.472613 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:24.472613 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:24.472614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:24.472971 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:24.473671 (+ 700us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:24.473756 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 15:20:24.473937 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":68,"threads_started":1}
I20260925 15:20:24.475526 6393 catalog_manager.cc:1347] Generated new cluster ID: ee81a3d23d1942eeb8501da036a4e647
I20260925 15:20:24.475616 6393 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:20:24.491889 6393 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:20:24.492522 6393 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:20:24.500224 6393 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17d8497cfa044ad582f23ad7a6d7129e: Generated new TSK 0
I20260925 15:20:24.500523 6393 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.211.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab.
Entry for principal kudu/127.3.211.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab.
I20260925 15:20:24.536845 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.193:0
--local_ip_for_outbound_sockets=127.3.211.193
--webserver_interface=127.3.211.193
--webserver_port=0
--tserver_master_addrs=127.3.211.254:35285
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab
--principal=kudu/127.3.211.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:36281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:20:24.640924 6406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:24.641157 6406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:24.641186 6406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:24.642493 6406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:24.642546 6406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:24.642578 6406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.193
I20260925 15:20:24.644261 6406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:36281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab
--principal=kudu/127.3.211.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.211.193
--webserver_port=0
--tserver_master_addrs=127.3.211.254:35285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.193
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:24.644590 6406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:24.644847 6406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:24.650259 6414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:24.650715 6412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:24.650691 6411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:24.650949 6406 server_base.cc:1034] running on GCE node
I20260925 15:20:24.651238 6406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:24.651521 6406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:24.652722 6406 hybrid_clock.cc:648] HybridClock initialized: now 1790349624652706 us; error 33 us; skew 500 ppm
Sep 25 15:20:24 dist-test-slave-4bk3 krb5kdc[6299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349624, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:24.655473 6406 init.cc:377] Logged in from keytab as kudu/127.3.211.193@KRBTEST.COM (short username kudu)
I20260925 15:20:24.656004 6406 webserver.cc:460] Webserver started at http://127.3.211.193:44151/ using document root <none> and password file <none>
I20260925 15:20:24.656272 6406 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:24.656322 6406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:24.656541 6406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:24.657697 6406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-0/data/instance:
uuid: "62c7ce8331c449ef82d6c2933b41b72f"
format_stamp: "Formatted at 2026-09-25 15:20:24 on dist-test-slave-4bk3"
server_key: "903f3c338fd01bd8bad2d89a78b9e930"
server_key_iv: "9d5dd68ebb2c61c28066b29f7a72aa63"
server_key_version: "encryptionkey@0"
I20260925 15:20:24.658103 6406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-0/wal/instance:
uuid: "62c7ce8331c449ef82d6c2933b41b72f"
format_stamp: "Formatted at 2026-09-25 15:20:24 on dist-test-slave-4bk3"
server_key: "903f3c338fd01bd8bad2d89a78b9e930"
server_key_iv: "9d5dd68ebb2c61c28066b29f7a72aa63"
server_key_version: "encryptionkey@0"
I20260925 15:20:24.659637 6406 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:20:24.660787 6421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:24.660988 6406 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 15:20:24.661080 6406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-0/wal
uuid: "62c7ce8331c449ef82d6c2933b41b72f"
format_stamp: "Formatted at 2026-09-25 15:20:24 on dist-test-slave-4bk3"
server_key: "903f3c338fd01bd8bad2d89a78b9e930"
server_key_iv: "9d5dd68ebb2c61c28066b29f7a72aa63"
server_key_version: "encryptionkey@0"
I20260925 15:20:24.661149 6406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:24.672420 6406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:24.675004 6406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:24.675127 6406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:24.675323 6406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:20:24.675678 6406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:20:24.675714 6406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:24.675760 6406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:20:24.675786 6406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:24.683162 6406 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.193:34055
I20260925 15:20:24.683243 6534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.193:34055 every 8 connection(s)
I20260925 15:20:24.683591 6406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
Sep 25 15:20:24 dist-test-slave-4bk3 krb5kdc[6299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349624, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.193@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:20:24.693301 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 6406
I20260925 15:20:24.693382 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-0/wal/instance
I20260925 15:20:24.693575 3919 external_mini_cluster.cc:1428] Setting key ba151619a5fa31f290f8f2b05293c31a
I20260925 15:20:24.694957 6383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:24.684611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:36781 (local address 127.3.211.254:35285)
0925 15:20:24.684697 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:24.684699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:24.684899 (+ 200us) server_negotiation.cc:407] Connection header received
0925 15:20:24.685443 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:24.685444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:24.685461 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:24.685567 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:24.686698 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:24.687283 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:24.688740 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:24.688975 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:24.689075 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:24.691490 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:24.691492 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:24.691492 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:24.691500 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:24.693042 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:24.693616 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:24.693617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:24.693617 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:24.693804 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:24.694158 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:24.694159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:24.694159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:24.694309 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:24.694393 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:24.694768 (+ 375us) server_negotiation.cc:299] Negotiation successful
0925 15:20:24.694863 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 15:20:24.695183 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:24.684545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:35285 (local address 127.3.211.193:36781)
0925 15:20:24.684731 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:24.684771 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:24.685254 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:24.685693 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:24.685697 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:24.686313 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:24.686533 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:24.686538 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:24.687402 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:24.687404 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:24.688567 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:24.688574 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:24.689119 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:24.689121 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:24.689266 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:24.689848 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:24.689857 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:24.691378 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:24.693143 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:24.693151 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:24.693153 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:24.693452 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:24.693891 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:24.693892 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:24.693893 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:24.694055 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:24.694428 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:24.694436 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:24.694621 (+ 185us) client_negotiation.cc:769] Sending connection context
0925 15:20:24.694746 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 15:20:24.694927 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":93,"threads_started":1}
I20260925 15:20:24.695698 6535 heartbeater.cc:347] Connected to a master server at 127.3.211.254:35285
I20260925 15:20:24.695806 6535 heartbeater.cc:464] Registering TS with master...
I20260925 15:20:24.695987 6535 heartbeater.cc:511] Master 127.3.211.254:35285 requested a full tablet report, sending...
I20260925 15:20:24.696532 6344 ts_manager.cc:194] Registered new tserver with Master: 62c7ce8331c449ef82d6c2933b41b72f (127.3.211.193:34055)
I20260925 15:20:24.697151 6344 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.193@KRBTEST.COM'} at 127.3.211.193:36781
WARNING: no policy specified for kudu/127.3.211.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab.
Entry for principal kudu/127.3.211.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab.
I20260925 15:20:24.721093 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.194:0
--local_ip_for_outbound_sockets=127.3.211.194
--webserver_interface=127.3.211.194
--webserver_port=0
--tserver_master_addrs=127.3.211.254:35285
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab
--principal=kudu/127.3.211.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:36281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:20:24.807406 6540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:24.807631 6540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:24.807662 6540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:24.808980 6540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:24.809031 6540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:24.809062 6540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.194
I20260925 15:20:24.810563 6540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:36281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab
--principal=kudu/127.3.211.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.211.194
--webserver_port=0
--tserver_master_addrs=127.3.211.254:35285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.194
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:24.810829 6540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:24.811079 6540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:24.816185 6545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:24.816324 6548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:24.816185 6546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:24.816505 6540 server_base.cc:1034] running on GCE node
I20260925 15:20:24.816754 6540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:24.817034 6540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:24.818233 6540 hybrid_clock.cc:648] HybridClock initialized: now 1790349624818224 us; error 36 us; skew 500 ppm
Sep 25 15:20:24 dist-test-slave-4bk3 krb5kdc[6299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349624, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:24.821173 6540 init.cc:377] Logged in from keytab as kudu/127.3.211.194@KRBTEST.COM (short username kudu)
I20260925 15:20:24.821714 6540 webserver.cc:460] Webserver started at http://127.3.211.194:44493/ using document root <none> and password file <none>
I20260925 15:20:24.821910 6540 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:24.821950 6540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:24.822045 6540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:24.823087 6540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-1/data/instance:
uuid: "d597c5c289a54fe59fa14f07be7c7a76"
format_stamp: "Formatted at 2026-09-25 15:20:24 on dist-test-slave-4bk3"
server_key: "878eaaa7e5d96b2c96e7a829132619bd"
server_key_iv: "265faaaecba74c6500cde516e288c1e9"
server_key_version: "encryptionkey@0"
I20260925 15:20:24.823398 6540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-1/wal/instance:
uuid: "d597c5c289a54fe59fa14f07be7c7a76"
format_stamp: "Formatted at 2026-09-25 15:20:24 on dist-test-slave-4bk3"
server_key: "878eaaa7e5d96b2c96e7a829132619bd"
server_key_iv: "265faaaecba74c6500cde516e288c1e9"
server_key_version: "encryptionkey@0"
I20260925 15:20:24.824820 6540 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:20:24.825649 6555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:24.825871 6540 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:24.825958 6540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-1/wal
uuid: "d597c5c289a54fe59fa14f07be7c7a76"
format_stamp: "Formatted at 2026-09-25 15:20:24 on dist-test-slave-4bk3"
server_key: "878eaaa7e5d96b2c96e7a829132619bd"
server_key_iv: "265faaaecba74c6500cde516e288c1e9"
server_key_version: "encryptionkey@0"
I20260925 15:20:24.826022 6540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:24.845491 6540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:24.847697 6540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:24.847815 6540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:24.848014 6540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:20:24.848275 6540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:20:24.848307 6540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:24.848395 6540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:20:24.848425 6540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:24.854609 6540 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.194:41419
I20260925 15:20:24.854704 6668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.194:41419 every 8 connection(s)
I20260925 15:20:24.855003 6540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
I20260925 15:20:24.856773 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 6540
I20260925 15:20:24.856874 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-1/wal/instance
I20260925 15:20:24.857071 3919 external_mini_cluster.cc:1428] Setting key ada4808dcff34106bccd8203390c3397
Sep 25 15:20:24 dist-test-slave-4bk3 krb5kdc[6299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349624, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.194@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.211.195@KRBTEST.COM; defaulting to no policy
I20260925 15:20:24.866375 6383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:24.855778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:59055 (local address 127.3.211.254:35285)
0925 15:20:24.855843 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:24.855845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:24.856057 (+ 212us) server_negotiation.cc:407] Connection header received
0925 15:20:24.856743 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:24.856745 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:24.856756 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:24.856813 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:24.858099 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:24.858579 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:24.859836 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:24.860049 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:24.860141 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:24.862756 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:24.862759 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:24.862759 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:24.862768 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:24.864763 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:24.865316 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:24.865317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:24.865318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:24.865390 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:24.865705 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:24.865706 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:24.865707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:24.865825 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:24.865884 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:24.866226 (+ 342us) server_negotiation.cc:299] Negotiation successful
0925 15:20:24.866305 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 15:20:24.866585 6671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:24.855759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:35285 (local address 127.3.211.194:59055)
0925 15:20:24.855938 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:24.855974 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:24.856600 (+ 626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:24.856905 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:24.856910 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:24.857719 (+ 809us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:24.857996 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:24.858003 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:24.858677 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:24.858678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:24.859722 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:24.859725 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:24.860153 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:24.860154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:24.860287 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:24.861128 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:24.861137 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:24.862631 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:24.864872 (+ 2241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:24.864883 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:24.864886 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:24.865196 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:24.865494 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:24.865495 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:24.865495 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:24.865628 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:24.865890 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:24.865895 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:24.866025 (+ 130us) client_negotiation.cc:769] Sending connection context
0925 15:20:24.866172 (+ 147us) client_negotiation.cc:240] Negotiation successful
0925 15:20:24.866347 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":96,"threads_started":1}
I20260925 15:20:24.867116 6669 heartbeater.cc:347] Connected to a master server at 127.3.211.254:35285
I20260925 15:20:24.867234 6669 heartbeater.cc:464] Registering TS with master...
I20260925 15:20:24.867513 6669 heartbeater.cc:511] Master 127.3.211.254:35285 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.195@KRBTEST.COM" created.
I20260925 15:20:24.867971 6344 ts_manager.cc:194] Registered new tserver with Master: d597c5c289a54fe59fa14f07be7c7a76 (127.3.211.194:41419)
I20260925 15:20:24.868503 6344 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.194@KRBTEST.COM'} at 127.3.211.194:59055
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab.
Entry for principal kudu/127.3.211.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab.
I20260925 15:20:24.887342 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.195:0
--local_ip_for_outbound_sockets=127.3.211.195
--webserver_interface=127.3.211.195
--webserver_port=0
--tserver_master_addrs=127.3.211.254:35285
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab
--principal=kudu/127.3.211.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:36281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:20:24.972412 6674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:24.972760 6674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:24.972812 6674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:24.974721 6674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:24.974800 6674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:24.974850 6674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.195
I20260925 15:20:24.976562 6674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:36281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab
--principal=kudu/127.3.211.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.211.195
--webserver_port=0
--tserver_master_addrs=127.3.211.254:35285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.195
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:24.976868 6674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:24.977129 6674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:24.982542 6679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:24.982546 6680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:24.982546 6682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:24.982752 6674 server_base.cc:1034] running on GCE node
I20260925 15:20:24.983081 6674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:24.983363 6674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:24.984571 6674 hybrid_clock.cc:648] HybridClock initialized: now 1790349624984539 us; error 46 us; skew 500 ppm
Sep 25 15:20:24 dist-test-slave-4bk3 krb5kdc[6299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349624, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:24.987250 6674 init.cc:377] Logged in from keytab as kudu/127.3.211.195@KRBTEST.COM (short username kudu)
I20260925 15:20:24.987859 6674 webserver.cc:460] Webserver started at http://127.3.211.195:37405/ using document root <none> and password file <none>
I20260925 15:20:24.988106 6674 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:24.988178 6674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:24.988397 6674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:24.989719 6674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-2/data/instance:
uuid: "c81721a636c64959886f663820c64d35"
format_stamp: "Formatted at 2026-09-25 15:20:24 on dist-test-slave-4bk3"
server_key: "afa2c0b04f97646e82de41fc186d2ed0"
server_key_iv: "81a829eadac00fe5e0ff3228f759b01b"
server_key_version: "encryptionkey@0"
I20260925 15:20:24.990155 6674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-2/wal/instance:
uuid: "c81721a636c64959886f663820c64d35"
format_stamp: "Formatted at 2026-09-25 15:20:24 on dist-test-slave-4bk3"
server_key: "afa2c0b04f97646e82de41fc186d2ed0"
server_key_iv: "81a829eadac00fe5e0ff3228f759b01b"
server_key_version: "encryptionkey@0"
I20260925 15:20:24.991719 6674 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:20:24.992763 6689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:24.993076 6674 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:24.993197 6674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-2/wal
uuid: "c81721a636c64959886f663820c64d35"
format_stamp: "Formatted at 2026-09-25 15:20:24 on dist-test-slave-4bk3"
server_key: "afa2c0b04f97646e82de41fc186d2ed0"
server_key_iv: "81a829eadac00fe5e0ff3228f759b01b"
server_key_version: "encryptionkey@0"
I20260925 15:20:24.993294 6674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:25.013538 6674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:25.016569 6674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:25.016841 6674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:25.017163 6674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:20:25.017526 6674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:20:25.017583 6674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:25.017629 6674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:20:25.017660 6674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:25.023900 6674 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.195:46085
I20260925 15:20:25.023936 6802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.195:46085 every 8 connection(s)
I20260925 15:20:25.024287 6674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
Sep 25 15:20:25 dist-test-slave-4bk3 krb5kdc[6299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349624, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.195@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:20:25.033227 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 6674
I20260925 15:20:25.033390 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-2/wal/instance
I20260925 15:20:25.033596 3919 external_mini_cluster.cc:1428] Setting key 8588ea9a65bd4e44a8f46bd6324704fa
I20260925 15:20:25.034423 6383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:25.025104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:46727 (local address 127.3.211.254:35285)
0925 15:20:25.025172 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:25.025174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:25.025394 (+ 220us) server_negotiation.cc:407] Connection header received
0925 15:20:25.025881 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:25.025881 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:25.025892 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:25.025962 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:25.026948 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:25.027478 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:25.028643 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:25.028864 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:25.028933 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:25.031273 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:25.031275 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:25.031279 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:25.031287 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:25.032820 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:25.033313 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:25.033314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:25.033315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:25.033377 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:25.033680 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:25.033681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:25.033681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:25.033841 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:25.033904 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:25.034235 (+ 331us) server_negotiation.cc:299] Negotiation successful
0925 15:20:25.034332 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 15:20:25.034655 6805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:25.025092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:35285 (local address 127.3.211.195:46727)
0925 15:20:25.025254 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:25.025299 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:25.025788 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:25.026019 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:25.026022 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:25.026617 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:25.026833 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:25.026838 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:25.027578 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:25.027578 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:25.028514 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:25.028517 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:25.028951 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:25.028951 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:25.029060 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:25.029633 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:25.029642 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:25.031151 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:25.032937 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:25.032946 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:25.032950 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:25.033217 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:25.033468 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:25.033469 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:25.033469 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:25.033593 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:25.033895 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:25.033901 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:25.034091 (+ 190us) client_negotiation.cc:769] Sending connection context
0925 15:20:25.034216 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 15:20:25.034392 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":91,"threads_started":1}
I20260925 15:20:25.035158 6803 heartbeater.cc:347] Connected to a master server at 127.3.211.254:35285
I20260925 15:20:25.035261 6803 heartbeater.cc:464] Registering TS with master...
I20260925 15:20:25.035475 6803 heartbeater.cc:511] Master 127.3.211.254:35285 requested a full tablet report, sending...
I20260925 15:20:25.035869 6344 ts_manager.cc:194] Registered new tserver with Master: c81721a636c64959886f663820c64d35 (127.3.211.195:46085)
I20260925 15:20:25.036226 6344 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.195@KRBTEST.COM'} at 127.3.211.195:46727
I20260925 15:20:25.045849 3919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 15:20:25.057530 6383 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:25.056439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52436 (local address 127.3.211.254:35285)
0925 15:20:25.056505 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:25.056507 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:25.056611 (+ 104us) server_negotiation.cc:407] Connection header received
0925 15:20:25.056657 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:25.056658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:25.056667 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:25.056750 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:25.057418 (+ 668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52436: BlockingRecv error: recv got EOF from 127.0.0.1:52436 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 15:20:25.058389 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 6406
I20260925 15:20:25.063231 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 6540
I20260925 15:20:25.067489 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 6674
I20260925 15:20:25.072016 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 6312
2026-09-25T15:20:25Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1947 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349610595785-3919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:20:25 dist-test-slave-4bk3 krb5kdc[6815](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:20:25 dist-test-slave-4bk3 krb5kdc[6815](info): set up 2 sockets
Sep 25 15:20:25 dist-test-slave-4bk3 krb5kdc[6815](info): commencing operation
krb5kdc: starting...
W20260925 15:20:26.144160 3919 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Sep 25 15:20:26 dist-test-slave-4bk3 krb5kdc[6815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:20:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:20:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.211.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
I20260925 15:20:26.273046 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:45643
--webserver_interface=127.3.211.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:45175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.211.254:45643
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:20:26.358702 6828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:26.358959 6828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:26.358986 6828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:26.360531 6828 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:20:26.360590 6828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:26.360605 6828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:26.360618 6828 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:20:26.360631 6828 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:20:26.362670 6828 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:45175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349610595785-3919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.211.254:45643
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:45643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.211.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:26.363037 6828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:26.363317 6828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:26.368881 6836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:26.369033 6828 server_base.cc:1034] running on GCE node
W20260925 15:20:26.368877 6834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:26.369131 6833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:26.369418 6828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:26.369786 6828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:26.370986 6828 hybrid_clock.cc:648] HybridClock initialized: now 1790349626370955 us; error 46 us; skew 500 ppm
Sep 25 15:20:26 dist-test-slave-4bk3 krb5kdc[6815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349626, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:26.373946 6828 init.cc:377] Logged in from keytab as kudu/127.3.211.254@KRBTEST.COM (short username kudu)
I20260925 15:20:26.374513 6828 webserver.cc:460] Webserver started at http://127.3.211.254:45105/ using document root <none> and password file <none>
I20260925 15:20:26.374783 6828 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:26.374835 6828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:26.374992 6828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:26.376556 6828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349610595785-3919-0/minicluster-data/master-0/data/instance:
uuid: "3ada4387feef49579b994aa2797a6ca5"
format_stamp: "Formatted at 2026-09-25 15:20:26 on dist-test-slave-4bk3"
server_key: "00d258fd0d839264745ea2b4f39882cf"
server_key_iv: "b3bbf57c918e9af62863848070d764b3"
server_key_version: "encryptionkey@0"
I20260925 15:20:26.376987 6828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349610595785-3919-0/minicluster-data/master-0/wal/instance:
uuid: "3ada4387feef49579b994aa2797a6ca5"
format_stamp: "Formatted at 2026-09-25 15:20:26 on dist-test-slave-4bk3"
server_key: "00d258fd0d839264745ea2b4f39882cf"
server_key_iv: "b3bbf57c918e9af62863848070d764b3"
server_key_version: "encryptionkey@0"
I20260925 15:20:26.378726 6828 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260925 15:20:26.379814 6843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:26.380131 6828 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:26.380231 6828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349610595785-3919-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349610595785-3919-0/minicluster-data/master-0/wal
uuid: "3ada4387feef49579b994aa2797a6ca5"
format_stamp: "Formatted at 2026-09-25 15:20:26 on dist-test-slave-4bk3"
server_key: "00d258fd0d839264745ea2b4f39882cf"
server_key_iv: "b3bbf57c918e9af62863848070d764b3"
server_key_version: "encryptionkey@0"
I20260925 15:20:26.380353 6828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349610595785-3919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349610595785-3919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349610595785-3919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:26.397634 6828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:26.402061 6828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:26.402259 6828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:26.406927 6828 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.254:45643
I20260925 15:20:26.407063 6895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.254:45643 every 8 connection(s)
I20260925 15:20:26.407509 6828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
I20260925 15:20:26.408910 6896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:26.408947 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 6828
I20260925 15:20:26.409092 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790349610595785-3919-0/minicluster-data/master-0/wal/instance
I20260925 15:20:26.409298 3919 external_mini_cluster.cc:1428] Setting key 2af872d727a9b84e5e74889ed9b2a8e5
I20260925 15:20:26.413905 6896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ada4387feef49579b994aa2797a6ca5: Bootstrap starting.
I20260925 15:20:26.414741 6896 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ada4387feef49579b994aa2797a6ca5: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:26.415079 6896 log.cc:824] T 00000000000000000000000000000000 P 3ada4387feef49579b994aa2797a6ca5: Log is configured to *not* fsync() on all Append() calls
I20260925 15:20:26.415978 6896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ada4387feef49579b994aa2797a6ca5: No bootstrap required, opened a new log
Sep 25 15:20:26 dist-test-slave-4bk3 krb5kdc[6815](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:20:26.418783 6896 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ada4387feef49579b994aa2797a6ca5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ada4387feef49579b994aa2797a6ca5" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 45643 } }
I20260925 15:20:26.418975 6896 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ada4387feef49579b994aa2797a6ca5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:26.419019 6896 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ada4387feef49579b994aa2797a6ca5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ada4387feef49579b994aa2797a6ca5, State: Initialized, Role: FOLLOWER
I20260925 15:20:26.419179 6896 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ada4387feef49579b994aa2797a6ca5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ada4387feef49579b994aa2797a6ca5" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 45643 } }
I20260925 15:20:26.419284 6896 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ada4387feef49579b994aa2797a6ca5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:20:26.419332 6896 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ada4387feef49579b994aa2797a6ca5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:20:26.419389 6896 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ada4387feef49579b994aa2797a6ca5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:26.420197 6896 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ada4387feef49579b994aa2797a6ca5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ada4387feef49579b994aa2797a6ca5" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 45643 } }
I20260925 15:20:26.420388 6896 leader_election.cc:302] T 00000000000000000000000000000000 P 3ada4387feef49579b994aa2797a6ca5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ada4387feef49579b994aa2797a6ca5; no voters:
I20260925 15:20:26.420620 6896 leader_election.cc:288] T 00000000000000000000000000000000 P 3ada4387feef49579b994aa2797a6ca5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:20:26.420657 6901 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ada4387feef49579b994aa2797a6ca5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:20:26.420774 6901 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ada4387feef49579b994aa2797a6ca5 [term 1 LEADER]: Becoming Leader. State: Replica: 3ada4387feef49579b994aa2797a6ca5, State: Running, Role: LEADER
I20260925 15:20:26.420892 6901 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ada4387feef49579b994aa2797a6ca5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ada4387feef49579b994aa2797a6ca5" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 45643 } }
I20260925 15:20:26.420943 6896 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ada4387feef49579b994aa2797a6ca5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:20:26.421267 6901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ada4387feef49579b994aa2797a6ca5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ada4387feef49579b994aa2797a6ca5. Latest consensus state: current_term: 1 leader_uuid: "3ada4387feef49579b994aa2797a6ca5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ada4387feef49579b994aa2797a6ca5" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 45643 } } }
I20260925 15:20:26.421355 6901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ada4387feef49579b994aa2797a6ca5 [sys.catalog]: This master's current role is: LEADER
I20260925 15:20:26.421643 6902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ada4387feef49579b994aa2797a6ca5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ada4387feef49579b994aa2797a6ca5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ada4387feef49579b994aa2797a6ca5" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 45643 } } }
I20260925 15:20:26.421833 6902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ada4387feef49579b994aa2797a6ca5 [sys.catalog]: This master's current role is: LEADER
W20260925 15:20:26.422860 6916 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3ada4387feef49579b994aa2797a6ca5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:20:26.422922 6916 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:20:26.422972 6907 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:20:26.423143 6907 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:20:26.424417 6899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:26.410149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49332 (local address 127.3.211.254:45643)
0925 15:20:26.410552 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:26.410566 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:26.410619 (+ 53us) server_negotiation.cc:407] Connection header received
0925 15:20:26.411540 (+ 921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:26.411555 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:26.411654 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:26.411885 (+ 231us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:26.412979 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:26.414411 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:26.415428 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:26.415945 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:26.416037 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:26.419084 (+ 3047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:26.419092 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:26.419092 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:26.419109 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:26.422237 (+ 3128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:26.422975 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:26.422979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:26.422979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:26.423102 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:26.423351 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:26.423353 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:26.423353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:26.423644 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:26.423806 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:26.423944 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 15:20:26.424112 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":176,"threads_started":1}
I20260925 15:20:26.424715 6907 catalog_manager.cc:1347] Generated new cluster ID: b153648e396b404f9980ecf439ffc7bb
I20260925 15:20:26.424774 6907 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:20:26.429086 3919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:20:26.432421 6907 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:20:26.433087 6907 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:20:26.437615 6907 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ada4387feef49579b994aa2797a6ca5: Generated new TSK 0
I20260925 15:20:26.437842 6907 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:20:26.447999 6899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:26.443347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49348 (local address 127.3.211.254:45643)
0925 15:20:26.443418 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:26.443424 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:26.443460 (+ 36us) server_negotiation.cc:407] Connection header received
0925 15:20:26.443564 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:26.443565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:26.443574 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:26.443643 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:26.444137 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:26.444808 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:26.445937 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:26.446178 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:26.446243 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:26.447324 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:26.447326 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:26.447326 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:20:26.447576 (+ 250us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:26.447593 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:20:26.447599 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:26.447651 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:26.447793 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 15:20:26.447825 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 15:20:26.451983 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 6828
2026-09-25T15:20:26Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1377 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:20:26 dist-test-slave-4bk3 krb5kdc[6929](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:20:26 dist-test-slave-4bk3 krb5kdc[6929](info): set up 2 sockets
Sep 25 15:20:26 dist-test-slave-4bk3 krb5kdc[6929](info): commencing operation
krb5kdc: starting...
W20260925 15:20:27.508426 3919 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Sep 25 15:20:27 dist-test-slave-4bk3 krb5kdc[6929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:20:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:20:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.211.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
I20260925 15:20:27.639778 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:41547
--webserver_interface=127.3.211.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:35199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.211.254:41547
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:20:27.729423 6942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:27.729691 6942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:27.729718 6942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:27.731055 6942 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:20:27.731114 6942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:27.731132 6942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:27.731144 6942 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:20:27.731156 6942 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:20:27.732952 6942 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:35199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.211.254:41547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:41547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.211.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:27.733240 6942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:27.733505 6942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:27.739006 6948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:27.738961 6947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:27.738961 6950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:27.739020 6942 server_base.cc:1034] running on GCE node
I20260925 15:20:27.739529 6942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:27.739876 6942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:27.741076 6942 hybrid_clock.cc:648] HybridClock initialized: now 1790349627741052 us; error 43 us; skew 500 ppm
Sep 25 15:20:27 dist-test-slave-4bk3 krb5kdc[6929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349627, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:27.743956 6942 init.cc:377] Logged in from keytab as kudu/127.3.211.254@KRBTEST.COM (short username kudu)
I20260925 15:20:27.744491 6942 webserver.cc:460] Webserver started at http://127.3.211.254:36329/ using document root <none> and password file <none>
I20260925 15:20:27.744824 6942 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:27.744889 6942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:27.745042 6942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:27.746140 6942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/master-0/data/instance:
uuid: "5ef993cd18ed48a68e7c4c08741b0abc"
format_stamp: "Formatted at 2026-09-25 15:20:27 on dist-test-slave-4bk3"
server_key: "6224ac0e76d7185145366849d4bcb7d1"
server_key_iv: "aad6e153aa0f343e2284fbfee7989ed9"
server_key_version: "encryptionkey@0"
I20260925 15:20:27.746454 6942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/master-0/wal/instance:
uuid: "5ef993cd18ed48a68e7c4c08741b0abc"
format_stamp: "Formatted at 2026-09-25 15:20:27 on dist-test-slave-4bk3"
server_key: "6224ac0e76d7185145366849d4bcb7d1"
server_key_iv: "aad6e153aa0f343e2284fbfee7989ed9"
server_key_version: "encryptionkey@0"
I20260925 15:20:27.747926 6942 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:20:27.748858 6957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:27.749058 6942 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 15:20:27.749159 6942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/master-0/wal
uuid: "5ef993cd18ed48a68e7c4c08741b0abc"
format_stamp: "Formatted at 2026-09-25 15:20:27 on dist-test-slave-4bk3"
server_key: "6224ac0e76d7185145366849d4bcb7d1"
server_key_iv: "aad6e153aa0f343e2284fbfee7989ed9"
server_key_version: "encryptionkey@0"
I20260925 15:20:27.749238 6942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:27.760116 6942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:27.762651 6942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:27.762784 6942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:27.766831 6942 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.254:41547
I20260925 15:20:27.767052 7009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.254:41547 every 8 connection(s)
I20260925 15:20:27.767307 6942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
I20260925 15:20:27.768502 7010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:27.770866 7010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ef993cd18ed48a68e7c4c08741b0abc: Bootstrap starting.
I20260925 15:20:27.771543 7010 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ef993cd18ed48a68e7c4c08741b0abc: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:27.771850 7010 log.cc:824] T 00000000000000000000000000000000 P 5ef993cd18ed48a68e7c4c08741b0abc: Log is configured to *not* fsync() on all Append() calls
I20260925 15:20:27.773026 7010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ef993cd18ed48a68e7c4c08741b0abc: No bootstrap required, opened a new log
I20260925 15:20:27.774756 7010 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ef993cd18ed48a68e7c4c08741b0abc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ef993cd18ed48a68e7c4c08741b0abc" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 41547 } }
I20260925 15:20:27.774932 7010 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ef993cd18ed48a68e7c4c08741b0abc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:27.774957 7010 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ef993cd18ed48a68e7c4c08741b0abc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ef993cd18ed48a68e7c4c08741b0abc, State: Initialized, Role: FOLLOWER
I20260925 15:20:27.775135 7010 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ef993cd18ed48a68e7c4c08741b0abc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ef993cd18ed48a68e7c4c08741b0abc" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 41547 } }
I20260925 15:20:27.775204 7010 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ef993cd18ed48a68e7c4c08741b0abc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:20:27.775228 7010 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ef993cd18ed48a68e7c4c08741b0abc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:20:27.775282 7010 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ef993cd18ed48a68e7c4c08741b0abc [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:27.775691 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 6942
I20260925 15:20:27.775831 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/master-0/wal/instance
I20260925 15:20:27.776005 3919 external_mini_cluster.cc:1428] Setting key 480e86245cfd327b6f1c4263fe969dfb
I20260925 15:20:27.776633 7010 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ef993cd18ed48a68e7c4c08741b0abc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ef993cd18ed48a68e7c4c08741b0abc" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 41547 } }
I20260925 15:20:27.776793 7010 leader_election.cc:302] T 00000000000000000000000000000000 P 5ef993cd18ed48a68e7c4c08741b0abc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ef993cd18ed48a68e7c4c08741b0abc; no voters:
I20260925 15:20:27.777002 7010 leader_election.cc:288] T 00000000000000000000000000000000 P 5ef993cd18ed48a68e7c4c08741b0abc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:20:27.777026 7015 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ef993cd18ed48a68e7c4c08741b0abc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:20:27.777145 7015 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ef993cd18ed48a68e7c4c08741b0abc [term 1 LEADER]: Becoming Leader. State: Replica: 5ef993cd18ed48a68e7c4c08741b0abc, State: Running, Role: LEADER
I20260925 15:20:27.777280 7010 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ef993cd18ed48a68e7c4c08741b0abc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:20:27.777284 7015 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ef993cd18ed48a68e7c4c08741b0abc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ef993cd18ed48a68e7c4c08741b0abc" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 41547 } }
I20260925 15:20:27.777827 7016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ef993cd18ed48a68e7c4c08741b0abc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ef993cd18ed48a68e7c4c08741b0abc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ef993cd18ed48a68e7c4c08741b0abc" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 41547 } } }
I20260925 15:20:27.777922 7016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ef993cd18ed48a68e7c4c08741b0abc [sys.catalog]: This master's current role is: LEADER
I20260925 15:20:27.778177 7017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ef993cd18ed48a68e7c4c08741b0abc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ef993cd18ed48a68e7c4c08741b0abc. Latest consensus state: current_term: 1 leader_uuid: "5ef993cd18ed48a68e7c4c08741b0abc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ef993cd18ed48a68e7c4c08741b0abc" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 41547 } } }
I20260925 15:20:27.778470 7017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ef993cd18ed48a68e7c4c08741b0abc [sys.catalog]: This master's current role is: LEADER
I20260925 15:20:27.778743 7023 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:20:27.779413 7023 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:20:27.781194 7023 catalog_manager.cc:1347] Generated new cluster ID: 02bd21ec42d84388ad9a446d0c92b549
I20260925 15:20:27.781252 7023 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 15:20:27 dist-test-slave-4bk3 krb5kdc[6929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:20:27.791254 7014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:27.776650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45480 (local address 127.3.211.254:41547)
0925 15:20:27.776874 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:27.776886 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:27.776919 (+ 33us) server_negotiation.cc:407] Connection header received
0925 15:20:27.777069 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:27.777076 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:27.777144 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:27.778012 (+ 868us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:27.778130 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:27.779546 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:27.780672 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:27.781447 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:27.781940 (+ 493us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:27.785137 (+ 3197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:27.785143 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:27.785143 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:27.785162 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:27.788850 (+ 3688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:27.789653 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:27.789658 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:27.789659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:27.789829 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:27.790068 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:27.790069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:27.790070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:27.790481 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:27.790660 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:27.790800 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 15:20:27.790976 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":112,"threads_started":1}
I20260925 15:20:27.796361 7023 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:20:27.796947 7023 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:20:27.801931 7023 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ef993cd18ed48a68e7c4c08741b0abc: Generated new TSK 0
I20260925 15:20:27.802122 7023 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.211.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab.
Entry for principal kudu/127.3.211.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab.
I20260925 15:20:27.835960 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.193:0
--local_ip_for_outbound_sockets=127.3.211.193
--webserver_interface=127.3.211.193
--webserver_port=0
--tserver_master_addrs=127.3.211.254:41547
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab
--principal=kudu/127.3.211.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:35199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:20:27.934942 7036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:27.935194 7036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:27.935246 7036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:27.936550 7036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:27.936628 7036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:27.936676 7036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.193
I20260925 15:20:27.938571 7036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:35199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab
--principal=kudu/127.3.211.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.211.193
--webserver_port=0
--tserver_master_addrs=127.3.211.254:41547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.193
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:27.938889 7036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:27.939146 7036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:27.944562 7042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:27.944562 7044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:27.944585 7041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:27.944801 7036 server_base.cc:1034] running on GCE node
I20260925 15:20:27.945024 7036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:27.945299 7036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:27.946487 7036 hybrid_clock.cc:648] HybridClock initialized: now 1790349627946473 us; error 30 us; skew 500 ppm
Sep 25 15:20:27 dist-test-slave-4bk3 krb5kdc[6929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349627, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:27.949170 7036 init.cc:377] Logged in from keytab as kudu/127.3.211.193@KRBTEST.COM (short username kudu)
I20260925 15:20:27.949641 7036 webserver.cc:460] Webserver started at http://127.3.211.193:42439/ using document root <none> and password file <none>
I20260925 15:20:27.949896 7036 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:27.949967 7036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:27.950115 7036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:27.951189 7036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-0/data/instance:
uuid: "156c359d2e6d45498d7dc3d98db5ab47"
format_stamp: "Formatted at 2026-09-25 15:20:27 on dist-test-slave-4bk3"
server_key: "7fc51a9120c7db15e9d0d7b08b2a2c9a"
server_key_iv: "258b7c76e28f0658f44cc323092851fb"
server_key_version: "encryptionkey@0"
I20260925 15:20:27.951596 7036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-0/wal/instance:
uuid: "156c359d2e6d45498d7dc3d98db5ab47"
format_stamp: "Formatted at 2026-09-25 15:20:27 on dist-test-slave-4bk3"
server_key: "7fc51a9120c7db15e9d0d7b08b2a2c9a"
server_key_iv: "258b7c76e28f0658f44cc323092851fb"
server_key_version: "encryptionkey@0"
I20260925 15:20:27.953159 7036 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:20:27.953953 7051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:27.954178 7036 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 15:20:27.954299 7036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-0/wal
uuid: "156c359d2e6d45498d7dc3d98db5ab47"
format_stamp: "Formatted at 2026-09-25 15:20:27 on dist-test-slave-4bk3"
server_key: "7fc51a9120c7db15e9d0d7b08b2a2c9a"
server_key_iv: "258b7c76e28f0658f44cc323092851fb"
server_key_version: "encryptionkey@0"
I20260925 15:20:27.954383 7036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:27.985992 7036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:27.988597 7036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:27.988797 7036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:27.989059 7036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:20:27.989405 7036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:20:27.989462 7036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:27.989497 7036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:20:27.989526 7036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:27.995679 7036 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.193:39763
I20260925 15:20:27.995718 7164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.193:39763 every 8 connection(s)
I20260925 15:20:27.996086 7036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
Sep 25 15:20:28 dist-test-slave-4bk3 krb5kdc[6929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349627, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.193@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:20:28.003371 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 7036
I20260925 15:20:28.003489 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-0/wal/instance
I20260925 15:20:28.003728 3919 external_mini_cluster.cc:1428] Setting key 55ef30bb0aedf13fc3fafd9aa10006b0
I20260925 15:20:28.007999 7014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:27.997091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:58691 (local address 127.3.211.254:41547)
0925 15:20:27.997138 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:27.997140 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:27.997433 (+ 293us) server_negotiation.cc:407] Connection header received
0925 15:20:27.997985 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:27.997986 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:27.998002 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:27.998068 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:27.999081 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:27.999609 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:28.000745 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:28.000980 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:28.001086 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:28.004003 (+ 2917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:28.004005 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:28.004005 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:28.004015 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:28.006151 (+ 2136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:28.006855 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:28.006857 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:28.006858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:28.006942 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:28.007257 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:28.007258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:28.007258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:28.007395 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:28.007473 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:28.007787 (+ 314us) server_negotiation.cc:299] Negotiation successful
0925 15:20:28.007902 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 15:20:28.008193 7167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:27.997057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:41547 (local address 127.3.211.193:58691)
0925 15:20:27.997277 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:27.997330 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:27.997874 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:27.998143 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:27.998148 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:27.998761 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:27.998967 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:27.998971 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:27.999721 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:27.999722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:28.000651 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:28.000654 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:28.001098 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:28.001099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:28.001240 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:28.001875 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:28.001884 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:28.003866 (+ 1982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:28.006301 (+ 2435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:28.006370 (+ 69us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:28.006383 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:28.006703 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:28.007028 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:28.007029 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:28.007030 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:28.007166 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:28.007472 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:28.007479 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:28.007645 (+ 166us) client_negotiation.cc:769] Sending connection context
0925 15:20:28.007765 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 15:20:28.007942 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":104,"threads_started":1}
I20260925 15:20:28.008719 7165 heartbeater.cc:347] Connected to a master server at 127.3.211.254:41547
I20260925 15:20:28.008846 7165 heartbeater.cc:464] Registering TS with master...
I20260925 15:20:28.009078 7165 heartbeater.cc:511] Master 127.3.211.254:41547 requested a full tablet report, sending...
I20260925 15:20:28.009765 6974 ts_manager.cc:194] Registered new tserver with Master: 156c359d2e6d45498d7dc3d98db5ab47 (127.3.211.193:39763)
I20260925 15:20:28.010443 6974 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.193@KRBTEST.COM'} at 127.3.211.193:58691
WARNING: no policy specified for kudu/127.3.211.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab.
Entry for principal kudu/127.3.211.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab.
I20260925 15:20:28.032178 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.194:0
--local_ip_for_outbound_sockets=127.3.211.194
--webserver_interface=127.3.211.194
--webserver_port=0
--tserver_master_addrs=127.3.211.254:41547
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab
--principal=kudu/127.3.211.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:35199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:20:28.120226 7170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:28.120484 7170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:28.120512 7170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:28.121699 7170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:28.121745 7170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:28.121775 7170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.194
I20260925 15:20:28.123279 7170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:35199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab
--principal=kudu/127.3.211.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.211.194
--webserver_port=0
--tserver_master_addrs=127.3.211.254:41547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.194
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:28.123502 7170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:28.123703 7170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:28.129192 7176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:28.129204 7175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:28.129462 7170 server_base.cc:1034] running on GCE node
W20260925 15:20:28.129212 7178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:28.129745 7170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:28.129982 7170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:28.131158 7170 hybrid_clock.cc:648] HybridClock initialized: now 1790349628131139 us; error 31 us; skew 500 ppm
Sep 25 15:20:28 dist-test-slave-4bk3 krb5kdc[6929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349628, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:28.134050 7170 init.cc:377] Logged in from keytab as kudu/127.3.211.194@KRBTEST.COM (short username kudu)
I20260925 15:20:28.134614 7170 webserver.cc:460] Webserver started at http://127.3.211.194:36849/ using document root <none> and password file <none>
I20260925 15:20:28.134883 7170 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:28.134958 7170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:28.135150 7170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:28.136300 7170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-1/data/instance:
uuid: "2fc292ed0aac451fadbe903de5f468ae"
format_stamp: "Formatted at 2026-09-25 15:20:28 on dist-test-slave-4bk3"
server_key: "be6607f5ed7fa742b65122b16a7939db"
server_key_iv: "6628858e68dc8fd34f219025e013ed8b"
server_key_version: "encryptionkey@0"
I20260925 15:20:28.136770 7170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-1/wal/instance:
uuid: "2fc292ed0aac451fadbe903de5f468ae"
format_stamp: "Formatted at 2026-09-25 15:20:28 on dist-test-slave-4bk3"
server_key: "be6607f5ed7fa742b65122b16a7939db"
server_key_iv: "6628858e68dc8fd34f219025e013ed8b"
server_key_version: "encryptionkey@0"
I20260925 15:20:28.138197 7170 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:28.139093 7185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:28.139279 7170 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:28.139397 7170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-1/wal
uuid: "2fc292ed0aac451fadbe903de5f468ae"
format_stamp: "Formatted at 2026-09-25 15:20:28 on dist-test-slave-4bk3"
server_key: "be6607f5ed7fa742b65122b16a7939db"
server_key_iv: "6628858e68dc8fd34f219025e013ed8b"
server_key_version: "encryptionkey@0"
I20260925 15:20:28.139480 7170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:28.161001 7170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:28.163539 7170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:28.163739 7170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:28.164013 7170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:20:28.164420 7170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:20:28.164476 7170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:28.164518 7170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:20:28.164547 7170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:28.171074 7170 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.194:41921
I20260925 15:20:28.171176 7298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.194:41921 every 8 connection(s)
I20260925 15:20:28.171602 7170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
I20260925 15:20:28.178514 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 7170
I20260925 15:20:28.178668 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-1/wal/instance
I20260925 15:20:28.178892 3919 external_mini_cluster.cc:1428] Setting key 944c2ddfc7558d689c7b089b405313f1
Sep 25 15:20:28 dist-test-slave-4bk3 krb5kdc[6929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349628, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.194@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.211.195@KRBTEST.COM; defaulting to no policy
I20260925 15:20:28.188923 7014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:28.172786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:52173 (local address 127.3.211.254:41547)
0925 15:20:28.172937 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:28.172939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:28.173230 (+ 291us) server_negotiation.cc:407] Connection header received
0925 15:20:28.173852 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:28.173853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:28.173868 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:28.173983 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:28.175188 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:28.175717 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:28.176991 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:28.177230 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:28.177312 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:28.181304 (+ 3992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:28.181308 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:28.181308 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:28.181324 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:28.187142 (+ 5818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:28.187778 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:28.187780 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:28.187781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:28.187852 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:28.188147 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:28.188148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:28.188149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:28.188284 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:28.188390 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:28.188717 (+ 327us) server_negotiation.cc:299] Negotiation successful
0925 15:20:28.188826 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260925 15:20:28.189153 7301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:28.172759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:41547 (local address 127.3.211.194:52173)
0925 15:20:28.173011 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:28.173065 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:28.173697 (+ 632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:28.174164 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:28.174169 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:28.174829 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:28.175045 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:28.175050 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:28.175837 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:28.175838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:28.176866 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:28.176869 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:28.177459 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:28.177461 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:28.177689 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:28.178617 (+ 928us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:28.178635 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:28.181150 (+ 2515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:28.187312 (+ 6162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:28.187324 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:28.187326 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:28.187690 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:28.187927 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:28.187928 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:28.187929 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:28.188052 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:28.188407 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:28.188414 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:28.188566 (+ 152us) client_negotiation.cc:769] Sending connection context
0925 15:20:28.188697 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 15:20:28.188880 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":166,"threads_started":1}
I20260925 15:20:28.189826 7299 heartbeater.cc:347] Connected to a master server at 127.3.211.254:41547
I20260925 15:20:28.189967 7299 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.195@KRBTEST.COM" created.
I20260925 15:20:28.190227 7299 heartbeater.cc:511] Master 127.3.211.254:41547 requested a full tablet report, sending...
I20260925 15:20:28.190716 6974 ts_manager.cc:194] Registered new tserver with Master: 2fc292ed0aac451fadbe903de5f468ae (127.3.211.194:41921)
I20260925 15:20:28.191129 6974 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.194@KRBTEST.COM'} at 127.3.211.194:52173
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab.
Entry for principal kudu/127.3.211.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab.
I20260925 15:20:28.207788 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.195:0
--local_ip_for_outbound_sockets=127.3.211.195
--webserver_interface=127.3.211.195
--webserver_port=0
--tserver_master_addrs=127.3.211.254:41547
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab
--principal=kudu/127.3.211.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:35199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:20:28.300493 7304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:28.300779 7304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:28.300827 7304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:28.302568 7304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:28.302652 7304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:28.302711 7304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.195
I20260925 15:20:28.305550 7304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:35199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab
--principal=kudu/127.3.211.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.211.195
--webserver_port=0
--tserver_master_addrs=127.3.211.254:41547
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.195
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:28.306034 7304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:28.306488 7304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:28.312297 7310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:28.312296 7309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:28.312477 7312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:28.312819 7304 server_base.cc:1034] running on GCE node
I20260925 15:20:28.313060 7304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:28.313349 7304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:28.314565 7304 hybrid_clock.cc:648] HybridClock initialized: now 1790349628314556 us; error 39 us; skew 500 ppm
Sep 25 15:20:28 dist-test-slave-4bk3 krb5kdc[6929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349628, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:28.317994 7304 init.cc:377] Logged in from keytab as kudu/127.3.211.195@KRBTEST.COM (short username kudu)
I20260925 15:20:28.318538 7304 webserver.cc:460] Webserver started at http://127.3.211.195:45067/ using document root <none> and password file <none>
I20260925 15:20:28.318814 7304 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:28.318894 7304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:28.319048 7304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:28.320149 7304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-2/data/instance:
uuid: "0965fbc024354f179ee29bc381c3fa70"
format_stamp: "Formatted at 2026-09-25 15:20:28 on dist-test-slave-4bk3"
server_key: "9fe3b57d842ccc3871f9951665adbb63"
server_key_iv: "625899471d3138c76577bdff1d300099"
server_key_version: "encryptionkey@0"
I20260925 15:20:28.320554 7304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-2/wal/instance:
uuid: "0965fbc024354f179ee29bc381c3fa70"
format_stamp: "Formatted at 2026-09-25 15:20:28 on dist-test-slave-4bk3"
server_key: "9fe3b57d842ccc3871f9951665adbb63"
server_key_iv: "625899471d3138c76577bdff1d300099"
server_key_version: "encryptionkey@0"
I20260925 15:20:28.322259 7304 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 15:20:28.323207 7319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:28.323469 7304 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:20:28.323589 7304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-2/wal
uuid: "0965fbc024354f179ee29bc381c3fa70"
format_stamp: "Formatted at 2026-09-25 15:20:28 on dist-test-slave-4bk3"
server_key: "9fe3b57d842ccc3871f9951665adbb63"
server_key_iv: "625899471d3138c76577bdff1d300099"
server_key_version: "encryptionkey@0"
I20260925 15:20:28.323676 7304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:28.333091 7304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:28.335569 7304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:28.335799 7304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:28.336074 7304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:20:28.336514 7304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:20:28.336576 7304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:28.336635 7304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:20:28.336666 7304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:28.343997 7304 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.195:37949
I20260925 15:20:28.344036 7432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.195:37949 every 8 connection(s)
I20260925 15:20:28.344482 7304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
Sep 25 15:20:28 dist-test-slave-4bk3 krb5kdc[6929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349628, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.195@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:20:28.354549 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 7304
I20260925 15:20:28.354732 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790349610595785-3919-0/minicluster-data/ts-2/wal/instance
I20260925 15:20:28.354960 3919 external_mini_cluster.cc:1428] Setting key b5c99f57ae06e6125bd3bf3c4f879149
I20260925 15:20:28.355808 7014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:28.345302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:45615 (local address 127.3.211.254:41547)
0925 15:20:28.345377 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:28.345379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:28.345582 (+ 203us) server_negotiation.cc:407] Connection header received
0925 15:20:28.346174 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:28.346175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:28.346189 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:28.346263 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:28.347293 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:28.347808 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:28.348963 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:28.349169 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:28.349269 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:28.352228 (+ 2959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:28.352235 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:28.352236 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:28.352245 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:28.353891 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:28.354481 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:28.354482 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:28.354483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:28.354555 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:28.354873 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:28.354875 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:28.354876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:28.355062 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:28.355178 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:28.355599 (+ 421us) server_negotiation.cc:299] Negotiation successful
0925 15:20:28.355710 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 15:20:28.355994 7435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:28.345285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:41547 (local address 127.3.211.195:45615)
0925 15:20:28.345445 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:28.345492 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:28.346052 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:28.346336 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:28.346341 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:28.346955 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:28.347168 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:28.347172 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:28.347913 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:28.347914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:28.348850 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:28.348852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:28.349282 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:28.349283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:28.349418 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:28.350076 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:28.350090 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:28.352075 (+ 1985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:28.354038 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:28.354061 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:28.354064 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:28.354388 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:28.354632 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:28.354633 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:28.354634 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:28.354800 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:28.355172 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:28.355179 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:28.355384 (+ 205us) client_negotiation.cc:769] Sending connection context
0925 15:20:28.355531 (+ 147us) client_negotiation.cc:240] Negotiation successful
0925 15:20:28.355737 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":81,"threads_started":1}
I20260925 15:20:28.356457 7433 heartbeater.cc:347] Connected to a master server at 127.3.211.254:41547
I20260925 15:20:28.356552 7433 heartbeater.cc:464] Registering TS with master...
I20260925 15:20:28.356760 7433 heartbeater.cc:511] Master 127.3.211.254:41547 requested a full tablet report, sending...
I20260925 15:20:28.357296 6974 ts_manager.cc:194] Registered new tserver with Master: 0965fbc024354f179ee29bc381c3fa70 (127.3.211.195:37949)
I20260925 15:20:28.357797 6974 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.195@KRBTEST.COM'} at 127.3.211.195:45615
I20260925 15:20:28.367888 3919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 15:20:28 dist-test-slave-4bk3 krb5kdc[6929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349628, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 25 15:20:28 dist-test-slave-4bk3 krb5kdc[6929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349628, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:20:28.395543 7014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:28.386432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45484 (local address 127.3.211.254:41547)
0925 15:20:28.386527 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:28.386530 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:28.386638 (+ 108us) server_negotiation.cc:407] Connection header received
0925 15:20:28.386885 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:28.386888 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:28.386910 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:28.387113 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:28.387755 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:28.388273 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:28.389078 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:28.389279 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:28.389371 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:28.392310 (+ 2939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:28.392311 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:28.392312 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:28.392320 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:28.394183 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:28.394622 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:28.394623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:28.394624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:28.394685 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:28.394915 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:28.394916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:28.394916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:28.395083 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:28.395197 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:28.395344 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 15:20:28.395433 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260925 15:20:28.396054 6974 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45484
I20260925 15:20:28.396800 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 7036
I20260925 15:20:28.401880 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 7170
I20260925 15:20:28.407070 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 7304
I20260925 15:20:28.411566 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 6942
2026-09-25T15:20:28Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1962 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:20:28 dist-test-slave-4bk3 krb5kdc[7445](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:20:28 dist-test-slave-4bk3 krb5kdc[7445](info): set up 2 sockets
Sep 25 15:20:28 dist-test-slave-4bk3 krb5kdc[7445](info): commencing operation
krb5kdc: starting...
W20260925 15:20:29.468575 3919 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Sep 25 15:20:29 dist-test-slave-4bk3 krb5kdc[7445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:20:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:20:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.211.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
I20260925 15:20:29.599684 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:33019
--webserver_interface=127.3.211.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:40171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.211.254:33019
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:20:29.684979 7458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:29.685263 7458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:29.685305 7458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:29.687507 7458 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:20:29.687600 7458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:29.687624 7458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:29.687646 7458 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:20:29.687686 7458 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:20:29.689627 7458 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:40171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.211.254:33019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:33019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.211.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:29.689980 7458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:29.690235 7458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:29.695669 7464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:29.695674 7466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:29.695744 7463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:29.695901 7458 server_base.cc:1034] running on GCE node
I20260925 15:20:29.696205 7458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:29.696568 7458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:29.697840 7458 hybrid_clock.cc:648] HybridClock initialized: now 1790349629697815 us; error 43 us; skew 500 ppm
Sep 25 15:20:29 dist-test-slave-4bk3 krb5kdc[7445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349629, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:29.700865 7458 init.cc:377] Logged in from keytab as kudu/127.3.211.254@KRBTEST.COM (short username kudu)
I20260925 15:20:29.701387 7458 webserver.cc:460] Webserver started at http://127.3.211.254:45805/ using document root <none> and password file <none>
I20260925 15:20:29.701629 7458 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:29.701699 7458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:29.701859 7458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:29.702957 7458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/master-0/data/instance:
uuid: "f46c67ec32a346ed857e075852505853"
format_stamp: "Formatted at 2026-09-25 15:20:29 on dist-test-slave-4bk3"
server_key: "4562325455a65f9f8834f5e92d4fead2"
server_key_iv: "35850af6c477f59d32b0ccc1e2386575"
server_key_version: "encryptionkey@0"
I20260925 15:20:29.703337 7458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/master-0/wal/instance:
uuid: "f46c67ec32a346ed857e075852505853"
format_stamp: "Formatted at 2026-09-25 15:20:29 on dist-test-slave-4bk3"
server_key: "4562325455a65f9f8834f5e92d4fead2"
server_key_iv: "35850af6c477f59d32b0ccc1e2386575"
server_key_version: "encryptionkey@0"
I20260925 15:20:29.705029 7458 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:20:29.705981 7473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:29.706211 7458 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:29.706321 7458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/master-0/wal
uuid: "f46c67ec32a346ed857e075852505853"
format_stamp: "Formatted at 2026-09-25 15:20:29 on dist-test-slave-4bk3"
server_key: "4562325455a65f9f8834f5e92d4fead2"
server_key_iv: "35850af6c477f59d32b0ccc1e2386575"
server_key_version: "encryptionkey@0"
I20260925 15:20:29.706413 7458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:29.725003 7458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:29.727178 7458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:29.727373 7458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:29.731827 7458 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.254:33019
I20260925 15:20:29.731900 7525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.254:33019 every 8 connection(s)
I20260925 15:20:29.732508 7458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
I20260925 15:20:29.733295 7526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:29.735525 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 7458
I20260925 15:20:29.735651 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/master-0/wal/instance
I20260925 15:20:29.735824 3919 external_mini_cluster.cc:1428] Setting key 6f48187e7f8c75b5a21edfc30765c0f8
I20260925 15:20:29.736142 7526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f46c67ec32a346ed857e075852505853: Bootstrap starting.
I20260925 15:20:29.736936 7526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f46c67ec32a346ed857e075852505853: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:29.737257 7526 log.cc:824] T 00000000000000000000000000000000 P f46c67ec32a346ed857e075852505853: Log is configured to *not* fsync() on all Append() calls
I20260925 15:20:29.738142 7526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f46c67ec32a346ed857e075852505853: No bootstrap required, opened a new log
I20260925 15:20:29.741003 7526 raft_consensus.cc:348] T 00000000000000000000000000000000 P f46c67ec32a346ed857e075852505853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46c67ec32a346ed857e075852505853" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 33019 } }
I20260925 15:20:29.741335 7526 raft_consensus.cc:374] T 00000000000000000000000000000000 P f46c67ec32a346ed857e075852505853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:29.741389 7526 raft_consensus.cc:729] T 00000000000000000000000000000000 P f46c67ec32a346ed857e075852505853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46c67ec32a346ed857e075852505853, State: Initialized, Role: FOLLOWER
I20260925 15:20:29.741537 7526 consensus_queue.cc:260] T 00000000000000000000000000000000 P f46c67ec32a346ed857e075852505853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46c67ec32a346ed857e075852505853" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 33019 } }
I20260925 15:20:29.741638 7526 raft_consensus.cc:388] T 00000000000000000000000000000000 P f46c67ec32a346ed857e075852505853 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:20:29.741683 7526 raft_consensus.cc:482] T 00000000000000000000000000000000 P f46c67ec32a346ed857e075852505853 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:20:29.741747 7526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f46c67ec32a346ed857e075852505853 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:29.742563 7526 raft_consensus.cc:504] T 00000000000000000000000000000000 P f46c67ec32a346ed857e075852505853 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46c67ec32a346ed857e075852505853" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 33019 } }
I20260925 15:20:29.742661 7526 leader_election.cc:302] T 00000000000000000000000000000000 P f46c67ec32a346ed857e075852505853 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f46c67ec32a346ed857e075852505853; no voters:
I20260925 15:20:29.742862 7526 leader_election.cc:288] T 00000000000000000000000000000000 P f46c67ec32a346ed857e075852505853 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:20:29.742936 7531 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f46c67ec32a346ed857e075852505853 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:20:29.743098 7531 raft_consensus.cc:686] T 00000000000000000000000000000000 P f46c67ec32a346ed857e075852505853 [term 1 LEADER]: Becoming Leader. State: Replica: f46c67ec32a346ed857e075852505853, State: Running, Role: LEADER
I20260925 15:20:29.743180 7526 sys_catalog.cc:564] T 00000000000000000000000000000000 P f46c67ec32a346ed857e075852505853 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:20:29.743237 7531 consensus_queue.cc:237] T 00000000000000000000000000000000 P f46c67ec32a346ed857e075852505853 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46c67ec32a346ed857e075852505853" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 33019 } }
I20260925 15:20:29.743597 7532 sys_catalog.cc:455] T 00000000000000000000000000000000 P f46c67ec32a346ed857e075852505853 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f46c67ec32a346ed857e075852505853" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46c67ec32a346ed857e075852505853" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 33019 } } }
I20260925 15:20:29.743593 7531 sys_catalog.cc:455] T 00000000000000000000000000000000 P f46c67ec32a346ed857e075852505853 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f46c67ec32a346ed857e075852505853. Latest consensus state: current_term: 1 leader_uuid: "f46c67ec32a346ed857e075852505853" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46c67ec32a346ed857e075852505853" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 33019 } } }
Sep 25 15:20:29 dist-test-slave-4bk3 krb5kdc[7445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349629, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:20:29.743925 7532 sys_catalog.cc:458] T 00000000000000000000000000000000 P f46c67ec32a346ed857e075852505853 [sys.catalog]: This master's current role is: LEADER
I20260925 15:20:29.743935 7531 sys_catalog.cc:458] T 00000000000000000000000000000000 P f46c67ec32a346ed857e075852505853 [sys.catalog]: This master's current role is: LEADER
I20260925 15:20:29.744176 7537 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:20:29.744720 7537 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:20:29.746564 7537 catalog_manager.cc:1347] Generated new cluster ID: 5e05dd1e8d4641e5853af7d09e5af84f
I20260925 15:20:29.746621 7537 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:20:29.748844 7529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:29.736520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45808 (local address 127.3.211.254:33019)
0925 15:20:29.736720 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:29.736728 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:29.736756 (+ 28us) server_negotiation.cc:407] Connection header received
0925 15:20:29.737281 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:29.737288 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:29.737379 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:29.737537 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:29.738331 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:29.739731 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:29.740660 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:29.741473 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:29.741607 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:29.744606 (+ 2999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:29.744612 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:29.744612 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:29.744639 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:29.746970 (+ 2331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:29.747476 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:29.747480 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:29.747480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:29.747585 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:29.747824 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:29.747825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:29.747825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:29.748131 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:29.748263 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:29.748464 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 15:20:29.748606 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":107,"threads_started":1}
I20260925 15:20:29.770500 7537 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:20:29.771193 7537 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:20:29.778760 7537 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f46c67ec32a346ed857e075852505853: Generated new TSK 0
I20260925 15:20:29.779022 7537 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.211.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab.
Entry for principal kudu/127.3.211.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab.
I20260925 15:20:29.815587 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.193:0
--local_ip_for_outbound_sockets=127.3.211.193
--webserver_interface=127.3.211.193
--webserver_port=0
--tserver_master_addrs=127.3.211.254:33019
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab
--principal=kudu/127.3.211.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:40171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:20:29.900852 7552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:29.901077 7552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:29.901104 7552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:29.902367 7552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:29.902427 7552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:29.902457 7552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.193
I20260925 15:20:29.904111 7552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:40171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab
--principal=kudu/127.3.211.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.211.193
--webserver_port=0
--tserver_master_addrs=127.3.211.254:33019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.193
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:29.904415 7552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:29.904698 7552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:29.909652 7557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:29.909685 7558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:29.909685 7560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:29.910059 7552 server_base.cc:1034] running on GCE node
I20260925 15:20:29.910337 7552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:29.910660 7552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:29.911855 7552 hybrid_clock.cc:648] HybridClock initialized: now 1790349629911844 us; error 33 us; skew 500 ppm
Sep 25 15:20:29 dist-test-slave-4bk3 krb5kdc[7445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349629, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:29.914525 7552 init.cc:377] Logged in from keytab as kudu/127.3.211.193@KRBTEST.COM (short username kudu)
I20260925 15:20:29.914994 7552 webserver.cc:460] Webserver started at http://127.3.211.193:34023/ using document root <none> and password file <none>
I20260925 15:20:29.915234 7552 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:29.915304 7552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:29.915462 7552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:29.916649 7552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/ts-0/data/instance:
uuid: "28241e0ee3a147e4b57b52ab63ecd051"
format_stamp: "Formatted at 2026-09-25 15:20:29 on dist-test-slave-4bk3"
server_key: "c9b90b542a385e30e6389dbb7c358e58"
server_key_iv: "4a8806081239c3cd00fd929a2e71d7e5"
server_key_version: "encryptionkey@0"
I20260925 15:20:29.917018 7552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/ts-0/wal/instance:
uuid: "28241e0ee3a147e4b57b52ab63ecd051"
format_stamp: "Formatted at 2026-09-25 15:20:29 on dist-test-slave-4bk3"
server_key: "c9b90b542a385e30e6389dbb7c358e58"
server_key_iv: "4a8806081239c3cd00fd929a2e71d7e5"
server_key_version: "encryptionkey@0"
I20260925 15:20:29.918442 7552 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:20:29.919314 7567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:29.919488 7552 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:29.919595 7552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/ts-0/wal
uuid: "28241e0ee3a147e4b57b52ab63ecd051"
format_stamp: "Formatted at 2026-09-25 15:20:29 on dist-test-slave-4bk3"
server_key: "c9b90b542a385e30e6389dbb7c358e58"
server_key_iv: "4a8806081239c3cd00fd929a2e71d7e5"
server_key_version: "encryptionkey@0"
I20260925 15:20:29.919682 7552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:29.930673 7552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:29.932958 7552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:29.933144 7552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:29.933408 7552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:20:29.933740 7552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:20:29.933797 7552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:29.933837 7552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:20:29.933880 7552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:29.940093 7552 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.193:39277
I20260925 15:20:29.940182 7680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.193:39277 every 8 connection(s)
I20260925 15:20:29.940549 7552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:20:29.941220 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 7552
I20260925 15:20:29.941323 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/ts-0/wal/instance
I20260925 15:20:29.941529 3919 external_mini_cluster.cc:1428] Setting key e393217e0012741acc12b791561fa472
Sep 25 15:20:29 dist-test-slave-4bk3 krb5kdc[7445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349629, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.193@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.211.194@KRBTEST.COM; defaulting to no policy
I20260925 15:20:29.952517 7529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:29.941826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:35295 (local address 127.3.211.254:33019)
0925 15:20:29.941931 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:29.941933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:29.941941 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:20:29.943071 (+ 1130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:29.943072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:29.943086 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:29.943153 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:29.943992 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:29.944800 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:29.946385 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:29.946587 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:29.946646 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:29.949431 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:29.949432 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:29.949433 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:29.949442 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:29.950883 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:29.951424 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:29.951426 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:29.951426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:29.951489 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:29.951775 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:29.951776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:29.951776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:29.951903 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:29.951981 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:29.952305 (+ 324us) server_negotiation.cc:299] Negotiation successful
0925 15:20:29.952419 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 15:20:29.952710 7683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:29.941502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:33019 (local address 127.3.211.193:35295)
0925 15:20:29.941700 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:29.941744 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:29.942969 (+ 1225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:29.943234 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:29.943239 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:29.943517 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:29.943853 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:29.943860 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:29.944890 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:29.944892 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:29.946293 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:29.946297 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:29.946694 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:29.946695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:29.946854 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:29.947432 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:29.947444 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:29.949305 (+ 1861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:29.950993 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:29.951002 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:29.951005 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:29.951332 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:29.951558 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:29.951559 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:29.951559 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:29.951698 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:29.951972 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:29.951979 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:29.952156 (+ 177us) client_negotiation.cc:769] Sending connection context
0925 15:20:29.952275 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 15:20:29.952464 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":90,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.194@KRBTEST.COM" created.
I20260925 15:20:29.953153 7681 heartbeater.cc:347] Connected to a master server at 127.3.211.254:33019
I20260925 15:20:29.953269 7681 heartbeater.cc:464] Registering TS with master...
I20260925 15:20:29.953464 7681 heartbeater.cc:511] Master 127.3.211.254:33019 requested a full tablet report, sending...
I20260925 15:20:29.954083 7490 ts_manager.cc:194] Registered new tserver with Master: 28241e0ee3a147e4b57b52ab63ecd051 (127.3.211.193:39277)
I20260925 15:20:29.954953 7490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.193@KRBTEST.COM'} at 127.3.211.193:35295
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab.
Entry for principal kudu/127.3.211.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab.
I20260925 15:20:29.968669 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.194:0
--local_ip_for_outbound_sockets=127.3.211.194
--webserver_interface=127.3.211.194
--webserver_port=0
--tserver_master_addrs=127.3.211.254:33019
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab
--principal=kudu/127.3.211.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:40171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:20:30.052304 7686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:30.052757 7686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:30.052810 7686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:30.055091 7686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:30.055243 7686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:30.055310 7686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.194
I20260925 15:20:30.057917 7686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:40171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab
--principal=kudu/127.3.211.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.211.194
--webserver_port=0
--tserver_master_addrs=127.3.211.254:33019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.194
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:30.058285 7686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:30.058568 7686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:30.063930 7692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:30.063938 7694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:30.063947 7691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:30.064284 7686 server_base.cc:1034] running on GCE node
I20260925 15:20:30.064589 7686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:30.064816 7686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:30.066041 7686 hybrid_clock.cc:648] HybridClock initialized: now 1790349630066006 us; error 66 us; skew 500 ppm
Sep 25 15:20:30 dist-test-slave-4bk3 krb5kdc[7445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349630, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:30.069154 7686 init.cc:377] Logged in from keytab as kudu/127.3.211.194@KRBTEST.COM (short username kudu)
I20260925 15:20:30.069671 7686 webserver.cc:460] Webserver started at http://127.3.211.194:41221/ using document root <none> and password file <none>
I20260925 15:20:30.069929 7686 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:30.070003 7686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:30.070158 7686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:30.071257 7686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/ts-1/data/instance:
uuid: "e28f1a349220428b917893accb69819e"
format_stamp: "Formatted at 2026-09-25 15:20:30 on dist-test-slave-4bk3"
server_key: "f4dba49c30289f7a928287ade3df6454"
server_key_iv: "8c66b5eac3f4c3d6fd7798ce1a00d986"
server_key_version: "encryptionkey@0"
I20260925 15:20:30.071631 7686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/ts-1/wal/instance:
uuid: "e28f1a349220428b917893accb69819e"
format_stamp: "Formatted at 2026-09-25 15:20:30 on dist-test-slave-4bk3"
server_key: "f4dba49c30289f7a928287ade3df6454"
server_key_iv: "8c66b5eac3f4c3d6fd7798ce1a00d986"
server_key_version: "encryptionkey@0"
I20260925 15:20:30.073100 7686 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:20:30.073971 7701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:30.074299 7686 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:20:30.074517 7686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/ts-1/wal
uuid: "e28f1a349220428b917893accb69819e"
format_stamp: "Formatted at 2026-09-25 15:20:30 on dist-test-slave-4bk3"
server_key: "f4dba49c30289f7a928287ade3df6454"
server_key_iv: "8c66b5eac3f4c3d6fd7798ce1a00d986"
server_key_version: "encryptionkey@0"
I20260925 15:20:30.074605 7686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:30.091394 7686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:30.093935 7686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:30.094112 7686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:30.094372 7686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:20:30.094686 7686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:20:30.094743 7686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:30.094816 7686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:20:30.094847 7686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:30.100879 7686 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.194:36735
I20260925 15:20:30.100919 7814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.194:36735 every 8 connection(s)
I20260925 15:20:30.101266 7686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
I20260925 15:20:30.104192 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 7686
I20260925 15:20:30.104380 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/ts-1/wal/instance
I20260925 15:20:30.104542 3919 external_mini_cluster.cc:1428] Setting key def18eb61a02b550b8a8ad87c9f54e7e
Sep 25 15:20:30 dist-test-slave-4bk3 krb5kdc[7445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349630, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.194@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:20:30.112247 7529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:30.102189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:51863 (local address 127.3.211.254:33019)
0925 15:20:30.102273 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:30.102275 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:30.102467 (+ 192us) server_negotiation.cc:407] Connection header received
0925 15:20:30.103458 (+ 991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:30.103460 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:30.103472 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:30.103525 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:30.104204 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:30.104954 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:30.106127 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:30.106410 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:30.106504 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:30.109262 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:30.109263 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:30.109264 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:30.109273 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:30.110797 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:30.111239 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:30.111240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:30.111241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:30.111301 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:30.111558 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:30.111559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:30.111560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:30.111687 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:30.111766 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:30.112055 (+ 289us) server_negotiation.cc:299] Negotiation successful
0925 15:20:30.112144 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 15:20:30.112525 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:30.102095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:33019 (local address 127.3.211.194:51863)
0925 15:20:30.102328 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:30.102377 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:30.103333 (+ 956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:30.103597 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:30.103601 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:30.103883 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:30.104103 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:30.104107 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:30.105051 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:30.105052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:30.105995 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:30.105998 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:30.106516 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:30.106517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:30.106658 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:30.107259 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:30.107279 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:30.109139 (+ 1860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:30.110898 (+ 1759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:30.110906 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:30.110909 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:30.111145 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:30.111370 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:30.111371 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:30.111371 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:30.111486 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:30.111757 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:30.111763 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:30.111911 (+ 148us) client_negotiation.cc:769] Sending connection context
0925 15:20:30.112021 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 15:20:30.112188 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":119,"threads_started":1}
I20260925 15:20:30.113059 7815 heartbeater.cc:347] Connected to a master server at 127.3.211.254:33019
I20260925 15:20:30.113196 7815 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.3.211.195@KRBTEST.COM; defaulting to no policy
I20260925 15:20:30.113399 7815 heartbeater.cc:511] Master 127.3.211.254:33019 requested a full tablet report, sending...
I20260925 15:20:30.113855 7490 ts_manager.cc:194] Registered new tserver with Master: e28f1a349220428b917893accb69819e (127.3.211.194:36735)
I20260925 15:20:30.114269 7490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.194@KRBTEST.COM'} at 127.3.211.194:51863
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab.
Entry for principal kudu/127.3.211.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab.
I20260925 15:20:30.131767 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.195:0
--local_ip_for_outbound_sockets=127.3.211.195
--webserver_interface=127.3.211.195
--webserver_port=0
--tserver_master_addrs=127.3.211.254:33019
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab
--principal=kudu/127.3.211.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:40171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:20:30.219009 7820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:30.219282 7820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:30.219326 7820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:30.220875 7820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:30.220950 7820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:30.221002 7820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.195
I20260925 15:20:30.222844 7820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:40171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab
--principal=kudu/127.3.211.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.211.195
--webserver_port=0
--tserver_master_addrs=127.3.211.254:33019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.195
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:30.223157 7820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:30.223423 7820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:30.228901 7826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:30.228888 7825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:30.229074 7820 server_base.cc:1034] running on GCE node
W20260925 15:20:30.228957 7828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:30.229423 7820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:30.229655 7820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:30.230823 7820 hybrid_clock.cc:648] HybridClock initialized: now 1790349630230808 us; error 30 us; skew 500 ppm
Sep 25 15:20:30 dist-test-slave-4bk3 krb5kdc[7445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349630, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:30.233856 7820 init.cc:377] Logged in from keytab as kudu/127.3.211.195@KRBTEST.COM (short username kudu)
I20260925 15:20:30.234356 7820 webserver.cc:460] Webserver started at http://127.3.211.195:45423/ using document root <none> and password file <none>
I20260925 15:20:30.234589 7820 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:30.234660 7820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:30.234817 7820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:30.235935 7820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/ts-2/data/instance:
uuid: "ba842256ff194711814ff09f26fda690"
format_stamp: "Formatted at 2026-09-25 15:20:30 on dist-test-slave-4bk3"
server_key: "00bc480fdc59e92cafd4d7bd1d1ec381"
server_key_iv: "d5f9670b379a3496ea0167c6d6f77678"
server_key_version: "encryptionkey@0"
I20260925 15:20:30.236323 7820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/ts-2/wal/instance:
uuid: "ba842256ff194711814ff09f26fda690"
format_stamp: "Formatted at 2026-09-25 15:20:30 on dist-test-slave-4bk3"
server_key: "00bc480fdc59e92cafd4d7bd1d1ec381"
server_key_iv: "d5f9670b379a3496ea0167c6d6f77678"
server_key_version: "encryptionkey@0"
I20260925 15:20:30.237972 7820 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:30.238871 7835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:30.239125 7820 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:30.239254 7820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/ts-2/wal
uuid: "ba842256ff194711814ff09f26fda690"
format_stamp: "Formatted at 2026-09-25 15:20:30 on dist-test-slave-4bk3"
server_key: "00bc480fdc59e92cafd4d7bd1d1ec381"
server_key_iv: "d5f9670b379a3496ea0167c6d6f77678"
server_key_version: "encryptionkey@0"
I20260925 15:20:30.239342 7820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:30.272709 7820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:30.275178 7820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:30.275360 7820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:30.275619 7820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:20:30.275949 7820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:20:30.276005 7820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:30.276043 7820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:20:30.276073 7820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:30.282886 7820 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.195:44073
I20260925 15:20:30.282980 7948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.195:44073 every 8 connection(s)
I20260925 15:20:30.283324 7820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
I20260925 15:20:30.287912 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 7820
I20260925 15:20:30.288031 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790349610595785-3919-0/minicluster-data/ts-2/wal/instance
I20260925 15:20:30.288221 3919 external_mini_cluster.cc:1428] Setting key 2a966225f673c30685fefd973734e9ab
Sep 25 15:20:30 dist-test-slave-4bk3 krb5kdc[7445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349630, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.195@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:20:30.294209 7529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:30.284191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:41449 (local address 127.3.211.254:33019)
0925 15:20:30.284235 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:30.284237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:30.284413 (+ 176us) server_negotiation.cc:407] Connection header received
0925 15:20:30.285422 (+ 1009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:30.285424 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:30.285434 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:30.285503 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:30.286205 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:30.286791 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:30.287968 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:30.288280 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:30.288413 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:30.291095 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:30.291097 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:30.291097 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:30.291105 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:30.292839 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:30.293281 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:30.293282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:30.293283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:30.293340 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:30.293567 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:30.293567 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:30.293568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:30.293695 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:30.293748 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:30.294073 (+ 325us) server_negotiation.cc:299] Negotiation successful
0925 15:20:30.294142 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 15:20:30.294482 7951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:30.284097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:33019 (local address 127.3.211.195:41449)
0925 15:20:30.284277 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:30.284321 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:30.285282 (+ 961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:30.285582 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:30.285585 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:30.285880 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:30.286083 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:30.286087 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:30.286916 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:30.286917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:30.287828 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:30.287831 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:30.288431 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:30.288433 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:30.288616 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:30.289305 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:30.289315 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:30.290967 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:30.292927 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:30.292935 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:30.292938 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:30.293204 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:30.293403 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:30.293404 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:30.293404 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:30.293510 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:30.293778 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:30.293789 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:30.293926 (+ 137us) client_negotiation.cc:769] Sending connection context
0925 15:20:30.294043 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 15:20:30.294191 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":113,"threads_started":1}
I20260925 15:20:30.295009 7949 heartbeater.cc:347] Connected to a master server at 127.3.211.254:33019
I20260925 15:20:30.295097 7949 heartbeater.cc:464] Registering TS with master...
I20260925 15:20:30.295245 7949 heartbeater.cc:511] Master 127.3.211.254:33019 requested a full tablet report, sending...
I20260925 15:20:30.295612 7490 ts_manager.cc:194] Registered new tserver with Master: ba842256ff194711814ff09f26fda690 (127.3.211.195:44073)
I20260925 15:20:30.295957 7490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.195@KRBTEST.COM'} at 127.3.211.195:41449
I20260925 15:20:30.300839 3919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 15:20:30 dist-test-slave-4bk3 krb5kdc[7445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 15:20:30 dist-test-slave-4bk3 krb5kdc[7445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.211.193@KRBTEST.COM
I20260925 15:20:30.328054 7956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:30.318661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40568 (local address 127.3.211.193:39277)
0925 15:20:30.318871 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:30.318876 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:30.318888 (+ 12us) server_negotiation.cc:407] Connection header received
0925 15:20:30.318915 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:30.318917 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:30.318931 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:30.319001 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:30.319641 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:30.320500 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:30.321638 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:30.322029 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:30.322094 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:30.324764 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:30.324773 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:30.324773 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:30.324781 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:30.326554 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:30.327146 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:30.327150 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:30.327150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:30.327234 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:30.327506 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:30.327507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:30.327507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:30.327649 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:30.327776 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:30.327904 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 15:20:30.327973 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":123,"threads_started":1}
Sep 25 15:20:30 dist-test-slave-4bk3 krb5kdc[7445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:20:30.336843 7529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:30.329633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45822 (local address 127.3.211.254:33019)
0925 15:20:30.329699 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:30.329701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:30.329711 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:20:30.329789 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:30.329790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:30.329801 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:30.329883 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:30.330393 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:30.330897 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:30.331608 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:30.331804 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:30.331882 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:30.334368 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:30.334370 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:30.334370 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:30.334377 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:30.335772 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:30.336108 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:30.336109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:30.336109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:30.336158 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:30.336393 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:30.336393 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:30.336394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:30.336540 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:30.336619 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:30.336718 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 15:20:30.336777 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260925 15:20:30.337090 7490 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45822
I20260925 15:20:30.337546 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 7552
I20260925 15:20:30.342301 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 7686
I20260925 15:20:30.347548 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 7820
I20260925 15:20:30.352429 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 7458
2026-09-25T15:20:30Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1939 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:20:30 dist-test-slave-4bk3 krb5kdc[7961](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:20:30 dist-test-slave-4bk3 krb5kdc[7961](info): set up 2 sockets
Sep 25 15:20:30 dist-test-slave-4bk3 krb5kdc[7961](info): commencing operation
krb5kdc: starting...
W20260925 15:20:31.464504 3919 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Sep 25 15:20:31 dist-test-slave-4bk3 krb5kdc[7961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:20:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:20:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.211.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
I20260925 15:20:31.593086 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:44069
--webserver_interface=127.3.211.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:36345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.211.254:44069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:20:31.683876 7974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:31.684158 7974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:31.684198 7974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:31.685474 7974 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:20:31.685559 7974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:31.685593 7974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:31.685627 7974 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:20:31.685649 7974 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:20:31.687202 7974 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:36345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.211.254:44069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:44069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.211.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:31.687505 7974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:31.687754 7974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:31.692663 7980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:31.692698 7982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:31.692633 7979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:31.692785 7974 server_base.cc:1034] running on GCE node
I20260925 15:20:31.693092 7974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:31.693411 7974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:31.694621 7974 hybrid_clock.cc:648] HybridClock initialized: now 1790349631694595 us; error 39 us; skew 500 ppm
Sep 25 15:20:31 dist-test-slave-4bk3 krb5kdc[7961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349631, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:31.697405 7974 init.cc:377] Logged in from keytab as kudu/127.3.211.254@KRBTEST.COM (short username kudu)
I20260925 15:20:31.697803 7974 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:31.697885 7974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:31.698061 7974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:31.699229 7974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/master-0/data/instance:
uuid: "bbfa2b772eff4111aee4fc4debc569f0"
format_stamp: "Formatted at 2026-09-25 15:20:31 on dist-test-slave-4bk3"
server_key: "0fa99a94f177ebd01f34783731e595b2"
server_key_iv: "42da6ea21d1de0d5db865e7ac4f82e0b"
server_key_version: "encryptionkey@0"
I20260925 15:20:31.699652 7974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/master-0/wal/instance:
uuid: "bbfa2b772eff4111aee4fc4debc569f0"
format_stamp: "Formatted at 2026-09-25 15:20:31 on dist-test-slave-4bk3"
server_key: "0fa99a94f177ebd01f34783731e595b2"
server_key_iv: "42da6ea21d1de0d5db865e7ac4f82e0b"
server_key_version: "encryptionkey@0"
I20260925 15:20:31.701272 7974 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:20:31.702153 7988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:31.702370 7974 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:31.702487 7974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/master-0/wal
uuid: "bbfa2b772eff4111aee4fc4debc569f0"
format_stamp: "Formatted at 2026-09-25 15:20:31 on dist-test-slave-4bk3"
server_key: "0fa99a94f177ebd01f34783731e595b2"
server_key_iv: "42da6ea21d1de0d5db865e7ac4f82e0b"
server_key_version: "encryptionkey@0"
I20260925 15:20:31.702571 7974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:31.710687 7974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:31.713186 7974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:31.713402 7974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:31.717757 7974 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.254:44069
I20260925 15:20:31.717814 8040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.254:44069 every 8 connection(s)
I20260925 15:20:31.718156 7974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
I20260925 15:20:31.718746 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 7974
I20260925 15:20:31.718865 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/master-0/wal/instance
I20260925 15:20:31.719074 3919 external_mini_cluster.cc:1428] Setting key 2583b0bedb5dc1fa351e521d1bcfbf98
I20260925 15:20:31.719280 8041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:31.722889 8041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbfa2b772eff4111aee4fc4debc569f0: Bootstrap starting.
I20260925 15:20:31.723605 8041 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbfa2b772eff4111aee4fc4debc569f0: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:31.723913 8041 log.cc:824] T 00000000000000000000000000000000 P bbfa2b772eff4111aee4fc4debc569f0: Log is configured to *not* fsync() on all Append() calls
I20260925 15:20:31.724774 8041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbfa2b772eff4111aee4fc4debc569f0: No bootstrap required, opened a new log
Sep 25 15:20:31 dist-test-slave-4bk3 krb5kdc[7961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:20:31.727145 8041 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbfa2b772eff4111aee4fc4debc569f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbfa2b772eff4111aee4fc4debc569f0" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 44069 } }
I20260925 15:20:31.727291 8041 raft_consensus.cc:374] T 00000000000000000000000000000000 P bbfa2b772eff4111aee4fc4debc569f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:31.727315 8041 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbfa2b772eff4111aee4fc4debc569f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbfa2b772eff4111aee4fc4debc569f0, State: Initialized, Role: FOLLOWER
I20260925 15:20:31.727481 8041 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbfa2b772eff4111aee4fc4debc569f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbfa2b772eff4111aee4fc4debc569f0" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 44069 } }
I20260925 15:20:31.727574 8041 raft_consensus.cc:388] T 00000000000000000000000000000000 P bbfa2b772eff4111aee4fc4debc569f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:20:31.727609 8041 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbfa2b772eff4111aee4fc4debc569f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:20:31.727664 8041 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbfa2b772eff4111aee4fc4debc569f0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:31.728540 8041 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbfa2b772eff4111aee4fc4debc569f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbfa2b772eff4111aee4fc4debc569f0" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 44069 } }
I20260925 15:20:31.728667 8041 leader_election.cc:302] T 00000000000000000000000000000000 P bbfa2b772eff4111aee4fc4debc569f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bbfa2b772eff4111aee4fc4debc569f0; no voters:
I20260925 15:20:31.728884 8041 leader_election.cc:288] T 00000000000000000000000000000000 P bbfa2b772eff4111aee4fc4debc569f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:20:31.728920 8046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbfa2b772eff4111aee4fc4debc569f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:20:31.729034 8046 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbfa2b772eff4111aee4fc4debc569f0 [term 1 LEADER]: Becoming Leader. State: Replica: bbfa2b772eff4111aee4fc4debc569f0, State: Running, Role: LEADER
I20260925 15:20:31.729126 8046 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbfa2b772eff4111aee4fc4debc569f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbfa2b772eff4111aee4fc4debc569f0" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 44069 } }
I20260925 15:20:31.729209 8041 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbfa2b772eff4111aee4fc4debc569f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:20:31.729451 8046 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbfa2b772eff4111aee4fc4debc569f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bbfa2b772eff4111aee4fc4debc569f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbfa2b772eff4111aee4fc4debc569f0" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 44069 } } }
I20260925 15:20:31.729519 8046 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbfa2b772eff4111aee4fc4debc569f0 [sys.catalog]: This master's current role is: LEADER
I20260925 15:20:31.729651 8046 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbfa2b772eff4111aee4fc4debc569f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbfa2b772eff4111aee4fc4debc569f0. Latest consensus state: current_term: 1 leader_uuid: "bbfa2b772eff4111aee4fc4debc569f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbfa2b772eff4111aee4fc4debc569f0" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 44069 } } }
I20260925 15:20:31.729720 8046 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbfa2b772eff4111aee4fc4debc569f0 [sys.catalog]: This master's current role is: LEADER
W20260925 15:20:31.730965 8062 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bbfa2b772eff4111aee4fc4debc569f0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:20:31.731024 8062 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:20:31.731076 8050 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:20:31.731297 8050 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:20:31.731449 8044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:31.719670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55722 (local address 127.3.211.254:44069)
0925 15:20:31.719879 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:31.719888 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:31.719927 (+ 39us) server_negotiation.cc:407] Connection header received
0925 15:20:31.720618 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:31.720629 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:31.720698 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:31.720875 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:31.721771 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:31.723126 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:31.724004 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:31.724637 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:31.724756 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:31.727412 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:31.727416 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:31.727417 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:31.727430 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:31.729556 (+ 2126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:31.730010 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:31.730013 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:31.730014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:31.730115 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:31.730358 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:31.730359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:31.730360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:31.730778 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:31.730944 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:31.731151 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 15:20:31.731301 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":95,"threads_started":1}
I20260925 15:20:31.732867 8050 catalog_manager.cc:1347] Generated new cluster ID: 141b90df3102478db3a4266e8262df43
I20260925 15:20:31.732923 8050 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:20:31.736420 8050 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:20:31.737205 8050 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:20:31.741520 8050 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bbfa2b772eff4111aee4fc4debc569f0: Generated new TSK 0
I20260925 15:20:31.741770 8050 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.211.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab.
Entry for principal kudu/127.3.211.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab.
I20260925 15:20:31.769762 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.193:0
--local_ip_for_outbound_sockets=127.3.211.193
--webserver_interface=127.3.211.193
--webserver_port=0
--tserver_master_addrs=127.3.211.254:44069
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab
--principal=kudu/127.3.211.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:36345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:20:31.859772 8067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:31.859999 8067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:31.860028 8067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:31.861374 8067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:31.861433 8067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:31.861464 8067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.193
I20260925 15:20:31.863102 8067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:36345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab
--principal=kudu/127.3.211.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.211.193
--webserver_port=0
--tserver_master_addrs=127.3.211.254:44069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.193
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:31.863399 8067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:31.863914 8067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:31.869227 8072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:31.869237 8075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:31.869264 8067 server_base.cc:1034] running on GCE node
W20260925 15:20:31.869237 8073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:31.869740 8067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:31.870005 8067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:31.871176 8067 hybrid_clock.cc:648] HybridClock initialized: now 1790349631871170 us; error 39 us; skew 500 ppm
Sep 25 15:20:31 dist-test-slave-4bk3 krb5kdc[7961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349631, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:31.874322 8067 init.cc:377] Logged in from keytab as kudu/127.3.211.193@KRBTEST.COM (short username kudu)
I20260925 15:20:31.874662 8067 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:31.874708 8067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:31.874804 8067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:31.875859 8067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/ts-0/data/instance:
uuid: "448890fe1d71481fb2706ef3976f7552"
format_stamp: "Formatted at 2026-09-25 15:20:31 on dist-test-slave-4bk3"
server_key: "e2ab8c1beecbf07b816385e780679e9b"
server_key_iv: "07b9daf6fcb30d2bbb16b28e5566c33a"
server_key_version: "encryptionkey@0"
I20260925 15:20:31.876171 8067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/ts-0/wal/instance:
uuid: "448890fe1d71481fb2706ef3976f7552"
format_stamp: "Formatted at 2026-09-25 15:20:31 on dist-test-slave-4bk3"
server_key: "e2ab8c1beecbf07b816385e780679e9b"
server_key_iv: "07b9daf6fcb30d2bbb16b28e5566c33a"
server_key_version: "encryptionkey@0"
I20260925 15:20:31.877529 8067 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:20:31.878193 8081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:31.878397 8067 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:20:31.878484 8067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/ts-0/wal
uuid: "448890fe1d71481fb2706ef3976f7552"
format_stamp: "Formatted at 2026-09-25 15:20:31 on dist-test-slave-4bk3"
server_key: "e2ab8c1beecbf07b816385e780679e9b"
server_key_iv: "07b9daf6fcb30d2bbb16b28e5566c33a"
server_key_version: "encryptionkey@0"
I20260925 15:20:31.878538 8067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:31.897449 8067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:31.899879 8067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:31.899998 8067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:31.900183 8067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:20:31.900528 8067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:20:31.900568 8067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:31.900591 8067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:20:31.900604 8067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:31.906790 8067 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.193:34465
I20260925 15:20:31.906885 8194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.193:34465 every 8 connection(s)
I20260925 15:20:31.907182 8067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
Sep 25 15:20:31 dist-test-slave-4bk3 krb5kdc[7961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349631, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.193@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:20:31.915791 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 8067
I20260925 15:20:31.915910 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/ts-0/wal/instance
I20260925 15:20:31.916110 3919 external_mini_cluster.cc:1428] Setting key c881a631c4e1da51ab49afcdaa4db4b1
I20260925 15:20:31.917716 8044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:31.907997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:41917 (local address 127.3.211.254:44069)
0925 15:20:31.908048 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:31.908051 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:31.908259 (+ 208us) server_negotiation.cc:407] Connection header received
0925 15:20:31.908844 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:31.908845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:31.908856 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:31.908927 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:31.909943 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:31.910471 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:31.911514 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:31.911745 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:31.911813 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:31.914260 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:31.914262 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:31.914262 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:31.914271 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:31.915834 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:31.916487 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:31.916489 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:31.916489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:31.916546 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:31.916912 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:31.916912 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:31.916913 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:31.917041 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:31.917108 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:31.917544 (+ 436us) server_negotiation.cc:299] Negotiation successful
0925 15:20:31.917633 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 15:20:31.917959 8197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:31.907921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:44069 (local address 127.3.211.193:41917)
0925 15:20:31.908098 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:31.908142 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:31.908727 (+ 585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:31.909008 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:31.909012 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:31.909292 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:31.909818 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:31.909823 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:31.910586 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:31.910587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:31.911413 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:31.911416 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:31.911852 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:31.911853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:31.911970 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:31.912607 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:31.912616 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:31.914110 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:31.915960 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:31.915971 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:31.915974 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:31.916302 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:31.916617 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:31.916618 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:31.916618 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:31.916774 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:31.917124 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:31.917132 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:31.917328 (+ 196us) client_negotiation.cc:769] Sending connection context
0925 15:20:31.917451 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 15:20:31.917656 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":99,"threads_started":1}
I20260925 15:20:31.918761 8195 heartbeater.cc:347] Connected to a master server at 127.3.211.254:44069
I20260925 15:20:31.918855 8195 heartbeater.cc:464] Registering TS with master...
I20260925 15:20:31.919129 8195 heartbeater.cc:511] Master 127.3.211.254:44069 requested a full tablet report, sending...
I20260925 15:20:31.919737 8005 ts_manager.cc:194] Registered new tserver with Master: 448890fe1d71481fb2706ef3976f7552 (127.3.211.193:34465)
I20260925 15:20:31.920480 8005 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.193@KRBTEST.COM'} at 127.3.211.193:41917
WARNING: no policy specified for kudu/127.3.211.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab.
Entry for principal kudu/127.3.211.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab.
I20260925 15:20:31.943588 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.194:0
--local_ip_for_outbound_sockets=127.3.211.194
--webserver_interface=127.3.211.194
--webserver_port=0
--tserver_master_addrs=127.3.211.254:44069
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab
--principal=kudu/127.3.211.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:36345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:20:32.032167 8200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:32.032670 8200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:32.032717 8200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:32.034832 8200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:32.034890 8200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:32.034921 8200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.194
I20260925 15:20:32.036798 8200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:36345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab
--principal=kudu/127.3.211.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.211.194
--webserver_port=0
--tserver_master_addrs=127.3.211.254:44069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.194
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:32.037041 8200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:32.037254 8200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:32.042609 8206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:32.042618 8205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:32.042630 8208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:32.043011 8200 server_base.cc:1034] running on GCE node
I20260925 15:20:32.043252 8200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:32.043514 8200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:32.044691 8200 hybrid_clock.cc:648] HybridClock initialized: now 1790349632044675 us; error 29 us; skew 500 ppm
Sep 25 15:20:32 dist-test-slave-4bk3 krb5kdc[7961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349632, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:32.047711 8200 init.cc:377] Logged in from keytab as kudu/127.3.211.194@KRBTEST.COM (short username kudu)
I20260925 15:20:32.048117 8200 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:32.048167 8200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:32.048309 8200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:32.049667 8200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/ts-1/data/instance:
uuid: "86978750d9ff4165b2d36f98b937a1b1"
format_stamp: "Formatted at 2026-09-25 15:20:32 on dist-test-slave-4bk3"
server_key: "451945d0ac814209555bf2aedf7eab87"
server_key_iv: "62dc22a5d87cdf96f14d8c5720d3adc2"
server_key_version: "encryptionkey@0"
I20260925 15:20:32.050396 8200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/ts-1/wal/instance:
uuid: "86978750d9ff4165b2d36f98b937a1b1"
format_stamp: "Formatted at 2026-09-25 15:20:32 on dist-test-slave-4bk3"
server_key: "451945d0ac814209555bf2aedf7eab87"
server_key_iv: "62dc22a5d87cdf96f14d8c5720d3adc2"
server_key_version: "encryptionkey@0"
I20260925 15:20:32.052302 8200 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260925 15:20:32.053680 8214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:32.054021 8200 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:32.054157 8200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/ts-1/wal
uuid: "86978750d9ff4165b2d36f98b937a1b1"
format_stamp: "Formatted at 2026-09-25 15:20:32 on dist-test-slave-4bk3"
server_key: "451945d0ac814209555bf2aedf7eab87"
server_key_iv: "62dc22a5d87cdf96f14d8c5720d3adc2"
server_key_version: "encryptionkey@0"
I20260925 15:20:32.054265 8200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:32.070528 8200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:32.073148 8200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:32.073335 8200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:32.073616 8200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:20:32.073971 8200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:20:32.074026 8200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:32.074070 8200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:20:32.074093 8200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:32.080557 8200 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.194:33237
I20260925 15:20:32.080623 8327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.194:33237 every 8 connection(s)
I20260925 15:20:32.080922 8200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
Sep 25 15:20:32 dist-test-slave-4bk3 krb5kdc[7961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349632, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.194@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:20:32.089609 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 8200
I20260925 15:20:32.089718 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/ts-1/wal/instance
I20260925 15:20:32.090052 3919 external_mini_cluster.cc:1428] Setting key 6f336ffa86ab68237f71d884f55481ad
I20260925 15:20:32.092805 8044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:32.081724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:42427 (local address 127.3.211.254:44069)
0925 15:20:32.081806 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:32.081808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:32.082012 (+ 204us) server_negotiation.cc:407] Connection header received
0925 15:20:32.082635 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:32.082637 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:32.082648 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:32.082721 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:32.083689 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:32.084188 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:32.085329 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:32.085522 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:32.085593 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:32.088945 (+ 3352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:32.088947 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:32.088947 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:32.088956 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:32.091059 (+ 2103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:32.091557 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:32.091559 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:32.091559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:32.091629 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:32.091900 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:32.091901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:32.091901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:32.092031 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:32.092111 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:32.092439 (+ 328us) server_negotiation.cc:299] Negotiation successful
0925 15:20:32.092690 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 15:20:32.092895 8330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:32.081695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:44069 (local address 127.3.211.194:42427)
0925 15:20:32.081871 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:32.081915 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:32.082524 (+ 609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:32.082780 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:32.082783 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:32.083056 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:32.083557 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:32.083562 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:32.084320 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:32.084321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:32.085206 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:32.085209 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:32.085639 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:32.085641 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:32.085770 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:32.086535 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:32.086550 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:32.088780 (+ 2230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:32.091168 (+ 2388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:32.091181 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:32.091184 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:32.091466 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:32.091698 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:32.091699 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:32.091700 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:32.091828 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:32.092100 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:32.092107 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:32.092270 (+ 163us) client_negotiation.cc:769] Sending connection context
0925 15:20:32.092418 (+ 148us) client_negotiation.cc:240] Negotiation successful
0925 15:20:32.092613 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":104,"threads_started":1}
I20260925 15:20:32.093314 8328 heartbeater.cc:347] Connected to a master server at 127.3.211.254:44069
I20260925 15:20:32.093430 8328 heartbeater.cc:464] Registering TS with master...
I20260925 15:20:32.093602 8328 heartbeater.cc:511] Master 127.3.211.254:44069 requested a full tablet report, sending...
I20260925 15:20:32.094020 8005 ts_manager.cc:194] Registered new tserver with Master: 86978750d9ff4165b2d36f98b937a1b1 (127.3.211.194:33237)
I20260925 15:20:32.094415 8005 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.194@KRBTEST.COM'} at 127.3.211.194:42427
WARNING: no policy specified for kudu/127.3.211.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab.
Entry for principal kudu/127.3.211.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab.
I20260925 15:20:32.118964 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.195:0
--local_ip_for_outbound_sockets=127.3.211.195
--webserver_interface=127.3.211.195
--webserver_port=0
--tserver_master_addrs=127.3.211.254:44069
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab
--principal=kudu/127.3.211.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:36345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:20:32.204706 8333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:32.204952 8333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:32.205019 8333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:32.206375 8333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:32.206424 8333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:32.206455 8333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.195
I20260925 15:20:32.208041 8333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:36345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab
--principal=kudu/127.3.211.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.211.195
--webserver_port=0
--tserver_master_addrs=127.3.211.254:44069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.195
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:32.208323 8333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:32.208597 8333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:32.214151 8338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:32.214177 8339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:32.214151 8341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:32.214191 8333 server_base.cc:1034] running on GCE node
I20260925 15:20:32.214784 8333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:32.215013 8333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:32.216216 8333 hybrid_clock.cc:648] HybridClock initialized: now 1790349632216208 us; error 46 us; skew 500 ppm
Sep 25 15:20:32 dist-test-slave-4bk3 krb5kdc[7961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349632, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:32.219208 8333 init.cc:377] Logged in from keytab as kudu/127.3.211.195@KRBTEST.COM (short username kudu)
I20260925 15:20:32.219599 8333 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:32.219673 8333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:32.219842 8333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:32.221000 8333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/ts-2/data/instance:
uuid: "6c726956bd29461aa3e494b8b126062a"
format_stamp: "Formatted at 2026-09-25 15:20:32 on dist-test-slave-4bk3"
server_key: "28860cebd73d821feab5686da16c130e"
server_key_iv: "0a81eb593641f48bf4c4bb934c693f7e"
server_key_version: "encryptionkey@0"
I20260925 15:20:32.221376 8333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/ts-2/wal/instance:
uuid: "6c726956bd29461aa3e494b8b126062a"
format_stamp: "Formatted at 2026-09-25 15:20:32 on dist-test-slave-4bk3"
server_key: "28860cebd73d821feab5686da16c130e"
server_key_iv: "0a81eb593641f48bf4c4bb934c693f7e"
server_key_version: "encryptionkey@0"
I20260925 15:20:32.222823 8333 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:20:32.223729 8347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:32.224044 8333 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:32.224148 8333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/ts-2/wal
uuid: "6c726956bd29461aa3e494b8b126062a"
format_stamp: "Formatted at 2026-09-25 15:20:32 on dist-test-slave-4bk3"
server_key: "28860cebd73d821feab5686da16c130e"
server_key_iv: "0a81eb593641f48bf4c4bb934c693f7e"
server_key_version: "encryptionkey@0"
I20260925 15:20:32.224231 8333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:32.240588 8333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:32.242966 8333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:32.243088 8333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:32.243304 8333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:20:32.243636 8333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:20:32.243670 8333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:32.243692 8333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:20:32.243706 8333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:32.249825 8333 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.195:41319
I20260925 15:20:32.249859 8460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.195:41319 every 8 connection(s)
I20260925 15:20:32.250142 8333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
I20260925 15:20:32.254516 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 8333
I20260925 15:20:32.254658 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/ts-2/wal/instance
I20260925 15:20:32.254875 3919 external_mini_cluster.cc:1428] Setting key 02ac26c1fd17a835c09f42478b463924
Sep 25 15:20:32 dist-test-slave-4bk3 krb5kdc[7961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349632, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.195@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:20:32.261268 8044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:32.250933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:37797 (local address 127.3.211.254:44069)
0925 15:20:32.250974 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:32.250978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:32.251186 (+ 208us) server_negotiation.cc:407] Connection header received
0925 15:20:32.251735 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:32.251736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:32.251748 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:32.251798 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:32.252853 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:32.253369 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:32.254454 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:32.254685 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:32.254755 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:32.257924 (+ 3169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:32.257925 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:32.257925 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:32.257933 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:32.259473 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:32.260058 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:32.260060 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:32.260060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:32.260122 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:32.260536 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:32.260537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:32.260537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:32.260669 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:32.260732 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:32.261109 (+ 377us) server_negotiation.cc:299] Negotiation successful
0925 15:20:32.261198 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 15:20:32.261603 8463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:32.250892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:44069 (local address 127.3.211.195:37797)
0925 15:20:32.251054 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:32.251096 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:32.251621 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:32.251868 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:32.251872 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:32.252150 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:32.252747 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:32.252754 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:32.253480 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:32.253481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:32.254332 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:32.254335 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:32.254799 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:32.254800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:32.254985 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:32.255911 (+ 926us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:32.255927 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:32.257792 (+ 1865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:32.259612 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:32.259622 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:32.259624 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:32.259959 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:32.260225 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:32.260226 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:32.260226 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:32.260460 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:32.260758 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:32.260765 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:32.260914 (+ 149us) client_negotiation.cc:769] Sending connection context
0925 15:20:32.261016 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 15:20:32.261285 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":72,"threads_started":1}
I20260925 15:20:32.262250 8461 heartbeater.cc:347] Connected to a master server at 127.3.211.254:44069
I20260925 15:20:32.262346 8461 heartbeater.cc:464] Registering TS with master...
I20260925 15:20:32.262472 8461 heartbeater.cc:511] Master 127.3.211.254:44069 requested a full tablet report, sending...
I20260925 15:20:32.262943 8005 ts_manager.cc:194] Registered new tserver with Master: 6c726956bd29461aa3e494b8b126062a (127.3.211.195:41319)
I20260925 15:20:32.263530 8005 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.195@KRBTEST.COM'} at 127.3.211.195:37797
I20260925 15:20:32.267853 3919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:20:32.277293 8044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:32.269740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60952 (local address 127.3.211.254:44069)
0925 15:20:32.269862 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:32.269864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:32.270139 (+ 275us) server_negotiation.cc:407] Connection header received
0925 15:20:32.270169 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:32.270170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:32.270182 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:32.270224 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:32.270994 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:32.271491 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:32.272436 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:32.272644 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:32.272723 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:32.274190 (+ 1467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:32.274191 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:32.274192 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:32.274199 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:32.275882 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:32.276282 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:32.276283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:32.276284 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:32.276409 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:32.276577 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:32.276577 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:32.276578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:32.276751 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:32.276836 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:32.277058 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 15:20:32.277167 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 15:20:32.279343 8005 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60952:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:20:32.281459 8005 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:20:32.288539 8474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:32.284298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:34465 (local address 127.0.0.1:38152)
0925 15:20:32.284618 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:32.284628 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:32.284671 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:32.285022 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:32.285026 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:32.285029 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:20:32.285106 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:32.285108 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:32.286119 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:32.286120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:32.287418 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:32.287421 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:32.288162 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:32.288164 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:32.288246 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:32.288261 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 15:20:32.288390 (+ 129us) client_negotiation.cc:240] Negotiation successful
0925 15:20:32.288453 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":4,"spinlock_wait_cycles":9088,"thread_start_us":136,"threads_started":1}
I20260925 15:20:32.288551 8476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:32.284629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38152 (local address 127.3.211.193:34465)
0925 15:20:32.284787 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:32.284793 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:32.284805 (+ 12us) server_negotiation.cc:407] Connection header received
0925 15:20:32.284848 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:32.284851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:32.284869 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:32.285188 (+ 319us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:20:32.285262 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:32.286007 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:32.287516 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:32.288040 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:32.288159 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:32.288225 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:32.288409 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 15:20:32.288454 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":96,"threads_started":1}
I20260925 15:20:32.288718 8473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:32.284306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.194:33237 (local address 127.0.0.1:35308)
0925 15:20:32.284746 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:32.284754 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:32.284789 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:32.285028 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:32.285029 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:32.285032 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:20:32.285112 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:32.285114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:32.286495 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:32.286496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:32.287568 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:32.287569 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:32.288452 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:32.288453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:32.288521 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:32.288525 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:20:32.288571 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 15:20:32.288622 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"mutex_wait_us":23,"thread_start_us":109,"threads_started":1}
I20260925 15:20:32.289139 8477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:32.284793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40316 (local address 127.3.211.195:41319)
0925 15:20:32.284941 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:32.284945 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:32.284955 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:20:32.285024 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:32.285026 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:32.285042 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:32.285121 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:20:32.285400 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:32.286451 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:32.287549 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:32.288299 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:32.288412 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:32.288471 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:32.288760 (+ 289us) server_negotiation.cc:299] Negotiation successful
0925 15:20:32.289058 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":77,"threads_started":1}
I20260925 15:20:32.289163 8475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:32.284429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35308 (local address 127.3.211.194:33237)
0925 15:20:32.284852 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:32.284856 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:32.284865 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:20:32.284903 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:32.284904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:32.284919 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:32.285000 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:20:32.285197 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:32.286386 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:32.287656 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:32.288359 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:32.288951 (+ 592us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:32.289020 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:32.289046 (+ 26us) server_negotiation.cc:299] Negotiation successful
0925 15:20:32.289089 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":79,"threads_started":1}
I20260925 15:20:32.288717 8472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:32.284298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.195:41319 (local address 127.0.0.1:40316)
0925 15:20:32.284522 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:32.284564 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:32.284642 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:32.285176 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:32.285177 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:32.285179 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:20:32.285223 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:32.285225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:32.286569 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:32.286570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:32.287448 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:32.287451 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:32.288498 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:32.288498 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:32.288556 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:32.288561 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:20:32.288608 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 15:20:32.288639 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":2,"thread_start_us":110,"threads_started":1}
I20260925 15:20:32.289482 8129 tablet_service.cc:1467] Processing CreateTablet for tablet ea9eb9fc08574a67bd8e2afe35c5e555 (DEFAULT_TABLE table=test-table [id=65b61ea73e334882b0bd2d08849cbc7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:20:32.289908 8129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea9eb9fc08574a67bd8e2afe35c5e555. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:32.289903 8395 tablet_service.cc:1467] Processing CreateTablet for tablet ea9eb9fc08574a67bd8e2afe35c5e555 (DEFAULT_TABLE table=test-table [id=65b61ea73e334882b0bd2d08849cbc7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:20:32.290235 8395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea9eb9fc08574a67bd8e2afe35c5e555. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:32.292740 8478 tablet_bootstrap.cc:492] T ea9eb9fc08574a67bd8e2afe35c5e555 P 448890fe1d71481fb2706ef3976f7552: Bootstrap starting.
I20260925 15:20:32.293136 8479 tablet_bootstrap.cc:492] T ea9eb9fc08574a67bd8e2afe35c5e555 P 6c726956bd29461aa3e494b8b126062a: Bootstrap starting.
I20260925 15:20:32.293159 8262 tablet_service.cc:1467] Processing CreateTablet for tablet ea9eb9fc08574a67bd8e2afe35c5e555 (DEFAULT_TABLE table=test-table [id=65b61ea73e334882b0bd2d08849cbc7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:20:32.293519 8262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea9eb9fc08574a67bd8e2afe35c5e555. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:32.293813 8478 tablet_bootstrap.cc:654] T ea9eb9fc08574a67bd8e2afe35c5e555 P 448890fe1d71481fb2706ef3976f7552: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:32.293813 8479 tablet_bootstrap.cc:654] T ea9eb9fc08574a67bd8e2afe35c5e555 P 6c726956bd29461aa3e494b8b126062a: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:32.294497 8478 log.cc:824] T ea9eb9fc08574a67bd8e2afe35c5e555 P 448890fe1d71481fb2706ef3976f7552: Log is configured to *not* fsync() on all Append() calls
I20260925 15:20:32.294497 8479 log.cc:824] T ea9eb9fc08574a67bd8e2afe35c5e555 P 6c726956bd29461aa3e494b8b126062a: Log is configured to *not* fsync() on all Append() calls
I20260925 15:20:32.295398 8479 tablet_bootstrap.cc:492] T ea9eb9fc08574a67bd8e2afe35c5e555 P 6c726956bd29461aa3e494b8b126062a: No bootstrap required, opened a new log
I20260925 15:20:32.295478 8479 ts_tablet_manager.cc:1397] T ea9eb9fc08574a67bd8e2afe35c5e555 P 6c726956bd29461aa3e494b8b126062a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:20:32.295552 8478 tablet_bootstrap.cc:492] T ea9eb9fc08574a67bd8e2afe35c5e555 P 448890fe1d71481fb2706ef3976f7552: No bootstrap required, opened a new log
I20260925 15:20:32.295619 8478 ts_tablet_manager.cc:1397] T ea9eb9fc08574a67bd8e2afe35c5e555 P 448890fe1d71481fb2706ef3976f7552: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 15:20:32.296792 8482 tablet_bootstrap.cc:492] T ea9eb9fc08574a67bd8e2afe35c5e555 P 86978750d9ff4165b2d36f98b937a1b1: Bootstrap starting.
I20260925 15:20:32.297384 8482 tablet_bootstrap.cc:654] T ea9eb9fc08574a67bd8e2afe35c5e555 P 86978750d9ff4165b2d36f98b937a1b1: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:32.297607 8478 raft_consensus.cc:348] T ea9eb9fc08574a67bd8e2afe35c5e555 P 448890fe1d71481fb2706ef3976f7552 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c726956bd29461aa3e494b8b126062a" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41319 } } peers { permanent_uuid: "86978750d9ff4165b2d36f98b937a1b1" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 33237 } } peers { permanent_uuid: "448890fe1d71481fb2706ef3976f7552" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34465 } }
I20260925 15:20:32.297595 8479 raft_consensus.cc:348] T ea9eb9fc08574a67bd8e2afe35c5e555 P 6c726956bd29461aa3e494b8b126062a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c726956bd29461aa3e494b8b126062a" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41319 } } peers { permanent_uuid: "86978750d9ff4165b2d36f98b937a1b1" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 33237 } } peers { permanent_uuid: "448890fe1d71481fb2706ef3976f7552" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34465 } }
I20260925 15:20:32.297771 8479 raft_consensus.cc:374] T ea9eb9fc08574a67bd8e2afe35c5e555 P 6c726956bd29461aa3e494b8b126062a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:32.297771 8478 raft_consensus.cc:374] T ea9eb9fc08574a67bd8e2afe35c5e555 P 448890fe1d71481fb2706ef3976f7552 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:32.297807 8479 raft_consensus.cc:729] T ea9eb9fc08574a67bd8e2afe35c5e555 P 6c726956bd29461aa3e494b8b126062a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c726956bd29461aa3e494b8b126062a, State: Initialized, Role: FOLLOWER
I20260925 15:20:32.297807 8478 raft_consensus.cc:729] T ea9eb9fc08574a67bd8e2afe35c5e555 P 448890fe1d71481fb2706ef3976f7552 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 448890fe1d71481fb2706ef3976f7552, State: Initialized, Role: FOLLOWER
I20260925 15:20:32.297987 8482 log.cc:824] T ea9eb9fc08574a67bd8e2afe35c5e555 P 86978750d9ff4165b2d36f98b937a1b1: Log is configured to *not* fsync() on all Append() calls
I20260925 15:20:32.297968 8479 consensus_queue.cc:260] T ea9eb9fc08574a67bd8e2afe35c5e555 P 6c726956bd29461aa3e494b8b126062a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c726956bd29461aa3e494b8b126062a" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41319 } } peers { permanent_uuid: "86978750d9ff4165b2d36f98b937a1b1" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 33237 } } peers { permanent_uuid: "448890fe1d71481fb2706ef3976f7552" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34465 } }
I20260925 15:20:32.297968 8478 consensus_queue.cc:260] T ea9eb9fc08574a67bd8e2afe35c5e555 P 448890fe1d71481fb2706ef3976f7552 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c726956bd29461aa3e494b8b126062a" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41319 } } peers { permanent_uuid: "86978750d9ff4165b2d36f98b937a1b1" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 33237 } } peers { permanent_uuid: "448890fe1d71481fb2706ef3976f7552" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34465 } }
I20260925 15:20:32.298273 8478 ts_tablet_manager.cc:1428] T ea9eb9fc08574a67bd8e2afe35c5e555 P 448890fe1d71481fb2706ef3976f7552: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:20:32.298273 8479 ts_tablet_manager.cc:1428] T ea9eb9fc08574a67bd8e2afe35c5e555 P 6c726956bd29461aa3e494b8b126062a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:20:32.298395 8195 heartbeater.cc:503] Master 127.3.211.254:44069 was elected leader, sending a full tablet report...
I20260925 15:20:32.298426 8461 heartbeater.cc:503] Master 127.3.211.254:44069 was elected leader, sending a full tablet report...
I20260925 15:20:32.299329 8482 tablet_bootstrap.cc:492] T ea9eb9fc08574a67bd8e2afe35c5e555 P 86978750d9ff4165b2d36f98b937a1b1: No bootstrap required, opened a new log
I20260925 15:20:32.299398 8482 ts_tablet_manager.cc:1397] T ea9eb9fc08574a67bd8e2afe35c5e555 P 86978750d9ff4165b2d36f98b937a1b1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 15:20:32.301136 8482 raft_consensus.cc:348] T ea9eb9fc08574a67bd8e2afe35c5e555 P 86978750d9ff4165b2d36f98b937a1b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c726956bd29461aa3e494b8b126062a" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41319 } } peers { permanent_uuid: "86978750d9ff4165b2d36f98b937a1b1" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 33237 } } peers { permanent_uuid: "448890fe1d71481fb2706ef3976f7552" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34465 } }
I20260925 15:20:32.301318 8482 raft_consensus.cc:374] T ea9eb9fc08574a67bd8e2afe35c5e555 P 86978750d9ff4165b2d36f98b937a1b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:32.301363 8482 raft_consensus.cc:729] T ea9eb9fc08574a67bd8e2afe35c5e555 P 86978750d9ff4165b2d36f98b937a1b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86978750d9ff4165b2d36f98b937a1b1, State: Initialized, Role: FOLLOWER
I20260925 15:20:32.301486 8482 consensus_queue.cc:260] T ea9eb9fc08574a67bd8e2afe35c5e555 P 86978750d9ff4165b2d36f98b937a1b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c726956bd29461aa3e494b8b126062a" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41319 } } peers { permanent_uuid: "86978750d9ff4165b2d36f98b937a1b1" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 33237 } } peers { permanent_uuid: "448890fe1d71481fb2706ef3976f7552" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34465 } }
I20260925 15:20:32.301759 8482 ts_tablet_manager.cc:1428] T ea9eb9fc08574a67bd8e2afe35c5e555 P 86978750d9ff4165b2d36f98b937a1b1: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 15:20:32.301818 8328 heartbeater.cc:503] Master 127.3.211.254:44069 was elected leader, sending a full tablet report...
W20260925 15:20:32.331530 8329 tablet.cc:2369] T ea9eb9fc08574a67bd8e2afe35c5e555 P 86978750d9ff4165b2d36f98b937a1b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:20:32.337471 8484 raft_consensus.cc:482] T ea9eb9fc08574a67bd8e2afe35c5e555 P 448890fe1d71481fb2706ef3976f7552 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:20:32.337639 8484 raft_consensus.cc:504] T ea9eb9fc08574a67bd8e2afe35c5e555 P 448890fe1d71481fb2706ef3976f7552 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c726956bd29461aa3e494b8b126062a" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41319 } } peers { permanent_uuid: "86978750d9ff4165b2d36f98b937a1b1" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 33237 } } peers { permanent_uuid: "448890fe1d71481fb2706ef3976f7552" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34465 } }
I20260925 15:20:32.338104 8484 leader_election.cc:288] T ea9eb9fc08574a67bd8e2afe35c5e555 P 448890fe1d71481fb2706ef3976f7552 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c726956bd29461aa3e494b8b126062a (127.3.211.195:41319), 86978750d9ff4165b2d36f98b937a1b1 (127.3.211.194:33237)
I20260925 15:20:32.342602 8477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:32.338424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:37479 (local address 127.3.211.195:41319)
0925 15:20:32.338473 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:32.338475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:32.338701 (+ 226us) server_negotiation.cc:407] Connection header received
0925 15:20:32.338761 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:32.338761 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:32.338769 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:32.338846 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:20:32.338983 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:32.339782 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:32.341506 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:32.342114 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:32.342204 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:32.342213 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:32.342477 (+ 264us) server_negotiation.cc:299] Negotiation successful
0925 15:20:32.342528 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 15:20:32.342602 8475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:32.338468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:47709 (local address 127.3.211.194:33237)
0925 15:20:32.338510 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:32.338513 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:32.338522 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:20:32.338558 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:32.338559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:32.338568 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:32.338648 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:20:32.338829 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:32.339748 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:32.341506 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:32.342117 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:32.342204 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:32.342213 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:32.342477 (+ 264us) server_negotiation.cc:299] Negotiation successful
0925 15:20:32.342528 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 15:20:32.342608 8197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:32.338335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.194:33237 (local address 127.3.211.193:47709)
0925 15:20:32.338399 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:32.338412 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:32.338468 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:32.338671 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:32.338671 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:32.338674 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:20:32.338753 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:32.338755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:32.339905 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:32.339907 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:32.341351 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:32.341355 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:32.342244 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:32.342246 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:32.342346 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:32.342348 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 15:20:32.342419 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 15:20:32.342476 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":121}
I20260925 15:20:32.342619 8487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:32.338340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.195:41319 (local address 127.3.211.193:37479)
0925 15:20:32.338594 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:32.338608 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:32.338681 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:32.338861 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:32.338864 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:32.338865 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:20:32.338916 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:32.338918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:32.339905 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:32.339907 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:32.341351 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:32.341355 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:32.342244 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:32.342246 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:32.342340 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:32.342345 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:20:32.342419 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 15:20:32.342476 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":125,"thread_start_us":127,"threads_started":1}
I20260925 15:20:32.343115 8282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea9eb9fc08574a67bd8e2afe35c5e555" candidate_uuid: "448890fe1d71481fb2706ef3976f7552" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86978750d9ff4165b2d36f98b937a1b1" is_pre_election: true
I20260925 15:20:32.343115 8415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea9eb9fc08574a67bd8e2afe35c5e555" candidate_uuid: "448890fe1d71481fb2706ef3976f7552" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c726956bd29461aa3e494b8b126062a" is_pre_election: true
I20260925 15:20:32.343293 8415 raft_consensus.cc:2454] T ea9eb9fc08574a67bd8e2afe35c5e555 P 6c726956bd29461aa3e494b8b126062a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 448890fe1d71481fb2706ef3976f7552 in term 0.
I20260925 15:20:32.343293 8282 raft_consensus.cc:2454] T ea9eb9fc08574a67bd8e2afe35c5e555 P 86978750d9ff4165b2d36f98b937a1b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 448890fe1d71481fb2706ef3976f7552 in term 0.
I20260925 15:20:32.343559 8085 leader_election.cc:302] T ea9eb9fc08574a67bd8e2afe35c5e555 P 448890fe1d71481fb2706ef3976f7552 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 448890fe1d71481fb2706ef3976f7552, 6c726956bd29461aa3e494b8b126062a; no voters:
I20260925 15:20:32.343726 8484 raft_consensus.cc:2781] T ea9eb9fc08574a67bd8e2afe35c5e555 P 448890fe1d71481fb2706ef3976f7552 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:20:32.343808 8484 raft_consensus.cc:482] T ea9eb9fc08574a67bd8e2afe35c5e555 P 448890fe1d71481fb2706ef3976f7552 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:20:32.343835 8484 raft_consensus.cc:3037] T ea9eb9fc08574a67bd8e2afe35c5e555 P 448890fe1d71481fb2706ef3976f7552 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:32.344630 8484 raft_consensus.cc:504] T ea9eb9fc08574a67bd8e2afe35c5e555 P 448890fe1d71481fb2706ef3976f7552 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c726956bd29461aa3e494b8b126062a" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41319 } } peers { permanent_uuid: "86978750d9ff4165b2d36f98b937a1b1" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 33237 } } peers { permanent_uuid: "448890fe1d71481fb2706ef3976f7552" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34465 } }
I20260925 15:20:32.344784 8484 leader_election.cc:288] T ea9eb9fc08574a67bd8e2afe35c5e555 P 448890fe1d71481fb2706ef3976f7552 [CANDIDATE]: Term 1 election: Requested vote from peers 6c726956bd29461aa3e494b8b126062a (127.3.211.195:41319), 86978750d9ff4165b2d36f98b937a1b1 (127.3.211.194:33237)
I20260925 15:20:32.344969 8415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea9eb9fc08574a67bd8e2afe35c5e555" candidate_uuid: "448890fe1d71481fb2706ef3976f7552" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c726956bd29461aa3e494b8b126062a"
I20260925 15:20:32.344969 8282 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea9eb9fc08574a67bd8e2afe35c5e555" candidate_uuid: "448890fe1d71481fb2706ef3976f7552" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86978750d9ff4165b2d36f98b937a1b1"
I20260925 15:20:32.345075 8415 raft_consensus.cc:3037] T ea9eb9fc08574a67bd8e2afe35c5e555 P 6c726956bd29461aa3e494b8b126062a [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:32.345070 8282 raft_consensus.cc:3037] T ea9eb9fc08574a67bd8e2afe35c5e555 P 86978750d9ff4165b2d36f98b937a1b1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:32.345817 8282 raft_consensus.cc:2454] T ea9eb9fc08574a67bd8e2afe35c5e555 P 86978750d9ff4165b2d36f98b937a1b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 448890fe1d71481fb2706ef3976f7552 in term 1.
I20260925 15:20:32.345820 8415 raft_consensus.cc:2454] T ea9eb9fc08574a67bd8e2afe35c5e555 P 6c726956bd29461aa3e494b8b126062a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 448890fe1d71481fb2706ef3976f7552 in term 1.
I20260925 15:20:32.346037 8084 leader_election.cc:302] T ea9eb9fc08574a67bd8e2afe35c5e555 P 448890fe1d71481fb2706ef3976f7552 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 448890fe1d71481fb2706ef3976f7552, 86978750d9ff4165b2d36f98b937a1b1; no voters:
I20260925 15:20:32.346170 8484 raft_consensus.cc:2781] T ea9eb9fc08574a67bd8e2afe35c5e555 P 448890fe1d71481fb2706ef3976f7552 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:20:32.346377 8484 raft_consensus.cc:686] T ea9eb9fc08574a67bd8e2afe35c5e555 P 448890fe1d71481fb2706ef3976f7552 [term 1 LEADER]: Becoming Leader. State: Replica: 448890fe1d71481fb2706ef3976f7552, State: Running, Role: LEADER
I20260925 15:20:32.346506 8484 consensus_queue.cc:237] T ea9eb9fc08574a67bd8e2afe35c5e555 P 448890fe1d71481fb2706ef3976f7552 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c726956bd29461aa3e494b8b126062a" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41319 } } peers { permanent_uuid: "86978750d9ff4165b2d36f98b937a1b1" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 33237 } } peers { permanent_uuid: "448890fe1d71481fb2706ef3976f7552" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34465 } }
I20260925 15:20:32.347515 8004 catalog_manager.cc:5495] T ea9eb9fc08574a67bd8e2afe35c5e555 P 448890fe1d71481fb2706ef3976f7552 reported cstate change: term changed from 0 to 1, leader changed from <none> to 448890fe1d71481fb2706ef3976f7552 (127.3.211.193). New cstate: current_term: 1 leader_uuid: "448890fe1d71481fb2706ef3976f7552" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c726956bd29461aa3e494b8b126062a" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86978750d9ff4165b2d36f98b937a1b1" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 33237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "448890fe1d71481fb2706ef3976f7552" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34465 } health_report { overall_health: HEALTHY } } }
I20260925 15:20:32.365252 8476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:32.362064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38164 (local address 127.3.211.193:34465)
0925 15:20:32.362168 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:32.362171 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:32.362188 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:20:32.362224 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:32.362225 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:32.362244 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:32.362325 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:20:32.362491 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:32.363189 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:32.364236 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:32.364549 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:32.364617 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:32.364802 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:20:32.364913 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:20:32.364981 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:32.365116 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 15:20:32.365143 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 15:20:32.367348 8282 raft_consensus.cc:1261] T ea9eb9fc08574a67bd8e2afe35c5e555 P 86978750d9ff4165b2d36f98b937a1b1 [term 1 FOLLOWER]: Refusing update from remote peer 448890fe1d71481fb2706ef3976f7552: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:20:32.367348 8415 raft_consensus.cc:1261] T ea9eb9fc08574a67bd8e2afe35c5e555 P 6c726956bd29461aa3e494b8b126062a [term 1 FOLLOWER]: Refusing update from remote peer 448890fe1d71481fb2706ef3976f7552: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:20:32.367779 8484 consensus_queue.cc:1035] T ea9eb9fc08574a67bd8e2afe35c5e555 P 448890fe1d71481fb2706ef3976f7552 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c726956bd29461aa3e494b8b126062a" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:20:32.367952 8488 consensus_queue.cc:1035] T ea9eb9fc08574a67bd8e2afe35c5e555 P 448890fe1d71481fb2706ef3976f7552 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86978750d9ff4165b2d36f98b937a1b1" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:20:32.371513 8493 mvcc.cc:205] Tried to move back new op lower bound from 7333272094173351936 to 7333272094091988992. Current Snapshot: MvccSnapshot[applied={T|T < 7333272094173351936}]
I20260925 15:20:32.373881 8004 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:20:32.374084 8004 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:20:32.375388 8004 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bbfa2b772eff4111aee4fc4debc569f0: Sending DeleteTablet for 3 replicas of tablet ea9eb9fc08574a67bd8e2afe35c5e555
I20260925 15:20:32.375854 8262 tablet_service.cc:1514] Processing DeleteTablet for tablet ea9eb9fc08574a67bd8e2afe35c5e555 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:20:32 UTC) from {username='kudu', principal='kudu/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:35308
I20260925 15:20:32.375881 8395 tablet_service.cc:1514] Processing DeleteTablet for tablet ea9eb9fc08574a67bd8e2afe35c5e555 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:20:32 UTC) from {username='kudu', principal='kudu/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:40316
I20260925 15:20:32.376149 8502 tablet_replica.cc:330] T ea9eb9fc08574a67bd8e2afe35c5e555 P 6c726956bd29461aa3e494b8b126062a: stopping tablet replica
I20260925 15:20:32.376165 8129 tablet_service.cc:1514] Processing DeleteTablet for tablet ea9eb9fc08574a67bd8e2afe35c5e555 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:20:32 UTC) from {username='kudu', principal='kudu/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:38152
I20260925 15:20:32.376273 8502 raft_consensus.cc:2229] T ea9eb9fc08574a67bd8e2afe35c5e555 P 6c726956bd29461aa3e494b8b126062a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:20:32.376421 8502 raft_consensus.cc:2258] T ea9eb9fc08574a67bd8e2afe35c5e555 P 6c726956bd29461aa3e494b8b126062a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:20:32.376741 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 8067
I20260925 15:20:32.376811 8503 tablet_replica.cc:330] T ea9eb9fc08574a67bd8e2afe35c5e555 P 448890fe1d71481fb2706ef3976f7552: stopping tablet replica
I20260925 15:20:32.376915 8503 raft_consensus.cc:2229] T ea9eb9fc08574a67bd8e2afe35c5e555 P 448890fe1d71481fb2706ef3976f7552 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:20:32.377033 8503 raft_consensus.cc:2258] T ea9eb9fc08574a67bd8e2afe35c5e555 P 448890fe1d71481fb2706ef3976f7552 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:20:32.377811 8502 ts_tablet_manager.cc:1905] T ea9eb9fc08574a67bd8e2afe35c5e555 P 6c726956bd29461aa3e494b8b126062a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:20:32.378931 8501 tablet_replica.cc:330] T ea9eb9fc08574a67bd8e2afe35c5e555 P 86978750d9ff4165b2d36f98b937a1b1: stopping tablet replica
I20260925 15:20:32.379094 8501 raft_consensus.cc:2229] T ea9eb9fc08574a67bd8e2afe35c5e555 P 86978750d9ff4165b2d36f98b937a1b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:20:32.379237 8501 raft_consensus.cc:2258] T ea9eb9fc08574a67bd8e2afe35c5e555 P 86978750d9ff4165b2d36f98b937a1b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:20:32.379729 8502 ts_tablet_manager.cc:1918] T ea9eb9fc08574a67bd8e2afe35c5e555 P 6c726956bd29461aa3e494b8b126062a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:20:32.379742 8501 ts_tablet_manager.cc:1905] T ea9eb9fc08574a67bd8e2afe35c5e555 P 86978750d9ff4165b2d36f98b937a1b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:20:32.379832 8502 log.cc:1196] T ea9eb9fc08574a67bd8e2afe35c5e555 P 6c726956bd29461aa3e494b8b126062a: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/ts-2/wal/wals/ea9eb9fc08574a67bd8e2afe35c5e555
I20260925 15:20:32.380105 8502 ts_tablet_manager.cc:1939] T ea9eb9fc08574a67bd8e2afe35c5e555 P 6c726956bd29461aa3e494b8b126062a: Deleting consensus metadata
I20260925 15:20:32.380779 7992 catalog_manager.cc:4841] TS 6c726956bd29461aa3e494b8b126062a (127.3.211.195:41319): tablet ea9eb9fc08574a67bd8e2afe35c5e555 (table test-table [id=65b61ea73e334882b0bd2d08849cbc7b]) successfully deleted
I20260925 15:20:32.381784 8501 ts_tablet_manager.cc:1918] T ea9eb9fc08574a67bd8e2afe35c5e555 P 86978750d9ff4165b2d36f98b937a1b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:20:32.381872 8501 log.cc:1196] T ea9eb9fc08574a67bd8e2afe35c5e555 P 86978750d9ff4165b2d36f98b937a1b1: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790349610595785-3919-0/minicluster-data/ts-1/wal/wals/ea9eb9fc08574a67bd8e2afe35c5e555
I20260925 15:20:32.382138 8501 ts_tablet_manager.cc:1939] T ea9eb9fc08574a67bd8e2afe35c5e555 P 86978750d9ff4165b2d36f98b937a1b1: Deleting consensus metadata
I20260925 15:20:32.382664 7991 catalog_manager.cc:4841] TS 86978750d9ff4165b2d36f98b937a1b1 (127.3.211.194:33237): tablet ea9eb9fc08574a67bd8e2afe35c5e555 (table test-table [id=65b61ea73e334882b0bd2d08849cbc7b]) successfully deleted
W20260925 15:20:32.383651 7990 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.211.193:34465 (error 108)
I20260925 15:20:32.383773 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 8200
I20260925 15:20:32.385215 8472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:32.385015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:34465 (local address 127.0.0.1:38176)
0925 15:20:32.385112 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:32.385175 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.193:34465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260925 15:20:32.385474 7990 catalog_manager.cc:4584] TS 448890fe1d71481fb2706ef3976f7552 (127.3.211.193:34465): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ea9eb9fc08574a67bd8e2afe35c5e555: Network error: Client connection negotiation failed: client connection to 127.3.211.193:34465: connect: Connection refused (error 111)
I20260925 15:20:32.389403 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 8333
I20260925 15:20:32.394369 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 7974
2026-09-25T15:20:32Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2041 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T15:20:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:20:32Z Disabled control of system clock
I20260925 15:20:32.423383 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:36785
--webserver_interface=127.3.211.254
--webserver_port=0
--builtin_ntp_servers=127.3.211.212:39405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.211.254:36785
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 15:20:32.509878 8508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:32.510102 8508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:32.510125 8508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:32.511399 8508 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:20:32.511456 8508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:32.511469 8508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:32.511482 8508 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:20:32.511495 8508 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:20:32.513015 8508 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:39405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.211.254:36785
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:36785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.211.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:32.513305 8508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:32.513643 8508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:32.518795 8514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:32.518756 8513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:32.519052 8516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:32.519157 8508 server_base.cc:1034] running on GCE node
I20260925 15:20:32.519425 8508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:32.519758 8508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:32.520938 8508 hybrid_clock.cc:648] HybridClock initialized: now 1790349632520930 us; error 48 us; skew 500 ppm
I20260925 15:20:32.522389 8508 webserver.cc:460] Webserver started at http://127.3.211.254:42369/ using document root <none> and password file <none>
I20260925 15:20:32.522662 8508 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:32.522738 8508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:32.522900 8508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:32.524032 8508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/master-0/data/instance:
uuid: "026e6f1ff967497eb3d8d2bedad98388"
format_stamp: "Formatted at 2026-09-25 15:20:32 on dist-test-slave-4bk3"
server_key: "dc7cd358448943ed28b95340d911c565"
server_key_iv: "35d1153aa944adce6b706dd04cd61969"
server_key_version: "encryptionkey@0"
I20260925 15:20:32.524485 8508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/master-0/wal/instance:
uuid: "026e6f1ff967497eb3d8d2bedad98388"
format_stamp: "Formatted at 2026-09-25 15:20:32 on dist-test-slave-4bk3"
server_key: "dc7cd358448943ed28b95340d911c565"
server_key_iv: "35d1153aa944adce6b706dd04cd61969"
server_key_version: "encryptionkey@0"
I20260925 15:20:32.525935 8508 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:20:32.526782 8522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:32.526991 8508 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:20:32.527098 8508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/master-0/wal
uuid: "026e6f1ff967497eb3d8d2bedad98388"
format_stamp: "Formatted at 2026-09-25 15:20:32 on dist-test-slave-4bk3"
server_key: "dc7cd358448943ed28b95340d911c565"
server_key_iv: "35d1153aa944adce6b706dd04cd61969"
server_key_version: "encryptionkey@0"
I20260925 15:20:32.527187 8508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:32.531052 8508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:32.531394 8508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:32.531550 8508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:32.535734 8508 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.254:36785
I20260925 15:20:32.535785 8574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.254:36785 every 8 connection(s)
I20260925 15:20:32.536191 8508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
I20260925 15:20:32.536971 8575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:32.538240 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 8508
I20260925 15:20:32.538355 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/master-0/wal/instance
I20260925 15:20:32.538543 3919 external_mini_cluster.cc:1428] Setting key f656f9726ea369c70293796af33bef4f
I20260925 15:20:32.541280 8575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 026e6f1ff967497eb3d8d2bedad98388: Bootstrap starting.
I20260925 15:20:32.541767 8578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:32.539285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53182 (local address 127.3.211.254:36785)
0925 15:20:32.539515 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:32.539526 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:32.539560 (+ 34us) server_negotiation.cc:407] Connection header received
0925 15:20:32.540240 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:32.540250 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:32.540360 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:32.540527 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:32.541271 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:32.541275 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:32.541275 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:20:32.541279 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:32.541327 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:20:32.541341 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:32.541420 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:32.541491 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 15:20:32.541548 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":88,"threads_started":1}
I20260925 15:20:32.542052 8575 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 026e6f1ff967497eb3d8d2bedad98388: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:32.542349 8575 log.cc:824] T 00000000000000000000000000000000 P 026e6f1ff967497eb3d8d2bedad98388: Log is configured to *not* fsync() on all Append() calls
I20260925 15:20:32.543313 8575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 026e6f1ff967497eb3d8d2bedad98388: No bootstrap required, opened a new log
I20260925 15:20:32.544898 8575 raft_consensus.cc:348] T 00000000000000000000000000000000 P 026e6f1ff967497eb3d8d2bedad98388 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "026e6f1ff967497eb3d8d2bedad98388" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 36785 } }
I20260925 15:20:32.545078 8575 raft_consensus.cc:374] T 00000000000000000000000000000000 P 026e6f1ff967497eb3d8d2bedad98388 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:32.545135 8575 raft_consensus.cc:729] T 00000000000000000000000000000000 P 026e6f1ff967497eb3d8d2bedad98388 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 026e6f1ff967497eb3d8d2bedad98388, State: Initialized, Role: FOLLOWER
I20260925 15:20:32.545262 8575 consensus_queue.cc:260] T 00000000000000000000000000000000 P 026e6f1ff967497eb3d8d2bedad98388 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "026e6f1ff967497eb3d8d2bedad98388" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 36785 } }
I20260925 15:20:32.545382 8575 raft_consensus.cc:388] T 00000000000000000000000000000000 P 026e6f1ff967497eb3d8d2bedad98388 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:20:32.545428 8575 raft_consensus.cc:482] T 00000000000000000000000000000000 P 026e6f1ff967497eb3d8d2bedad98388 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:20:32.545485 8575 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 026e6f1ff967497eb3d8d2bedad98388 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:32.546106 8575 raft_consensus.cc:504] T 00000000000000000000000000000000 P 026e6f1ff967497eb3d8d2bedad98388 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "026e6f1ff967497eb3d8d2bedad98388" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 36785 } }
I20260925 15:20:32.546233 8575 leader_election.cc:302] T 00000000000000000000000000000000 P 026e6f1ff967497eb3d8d2bedad98388 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 026e6f1ff967497eb3d8d2bedad98388; no voters:
I20260925 15:20:32.546434 8575 leader_election.cc:288] T 00000000000000000000000000000000 P 026e6f1ff967497eb3d8d2bedad98388 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:20:32.546746 8575 sys_catalog.cc:564] T 00000000000000000000000000000000 P 026e6f1ff967497eb3d8d2bedad98388 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:20:32.546798 8580 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 026e6f1ff967497eb3d8d2bedad98388 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:20:32.546959 8580 raft_consensus.cc:686] T 00000000000000000000000000000000 P 026e6f1ff967497eb3d8d2bedad98388 [term 1 LEADER]: Becoming Leader. State: Replica: 026e6f1ff967497eb3d8d2bedad98388, State: Running, Role: LEADER
I20260925 15:20:32.547060 8580 consensus_queue.cc:237] T 00000000000000000000000000000000 P 026e6f1ff967497eb3d8d2bedad98388 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "026e6f1ff967497eb3d8d2bedad98388" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 36785 } }
I20260925 15:20:32.547479 8581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 026e6f1ff967497eb3d8d2bedad98388 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "026e6f1ff967497eb3d8d2bedad98388" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "026e6f1ff967497eb3d8d2bedad98388" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 36785 } } }
I20260925 15:20:32.547478 8582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 026e6f1ff967497eb3d8d2bedad98388 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 026e6f1ff967497eb3d8d2bedad98388. Latest consensus state: current_term: 1 leader_uuid: "026e6f1ff967497eb3d8d2bedad98388" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "026e6f1ff967497eb3d8d2bedad98388" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 36785 } } }
I20260925 15:20:32.547600 8581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 026e6f1ff967497eb3d8d2bedad98388 [sys.catalog]: This master's current role is: LEADER
I20260925 15:20:32.547607 8582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 026e6f1ff967497eb3d8d2bedad98388 [sys.catalog]: This master's current role is: LEADER
I20260925 15:20:32.547809 8589 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:20:32.548318 8589 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:20:32.550025 8589 catalog_manager.cc:1347] Generated new cluster ID: 1168c92113544d388bb53cc9fc49da64
I20260925 15:20:32.550086 8589 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:20:32.566462 8589 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:20:32.566686 8589 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:20:32.576363 8589 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 026e6f1ff967497eb3d8d2bedad98388: Generated new TSK 0
I20260925 15:20:32.576604 8589 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:20:32.587785 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.193:0
--local_ip_for_outbound_sockets=127.3.211.193
--webserver_interface=127.3.211.193
--webserver_port=0
--tserver_master_addrs=127.3.211.254:36785
--builtin_ntp_servers=127.3.211.212:39405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 15:20:32.682628 8599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:32.682919 8599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:32.682960 8599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:32.684404 8599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:32.684484 8599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:32.684532 8599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.193
I20260925 15:20:32.686075 8599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:39405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.211.193
--webserver_port=0
--tserver_master_addrs=127.3.211.254:36785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.193
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:32.686405 8599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:32.686748 8599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:32.691928 8607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:32.692605 8604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:32.692812 8599 server_base.cc:1034] running on GCE node
W20260925 15:20:32.692822 8605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:32.693109 8599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:32.693410 8599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:32.694638 8599 hybrid_clock.cc:648] HybridClock initialized: now 1790349632694571 us; error 91 us; skew 500 ppm
I20260925 15:20:32.696158 8599 webserver.cc:460] Webserver started at http://127.3.211.193:43481/ using document root <none> and password file <none>
I20260925 15:20:32.696503 8599 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:32.696586 8599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:32.696753 8599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:32.698081 8599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/ts-0/data/instance:
uuid: "c170c1bf9b31476b9fcef7c83c46b0b6"
format_stamp: "Formatted at 2026-09-25 15:20:32 on dist-test-slave-4bk3"
server_key: "f7b8db00634d1997f7893fd191c83adf"
server_key_iv: "12c7c8b1b5a684fbd7fd58f2160c9bc4"
server_key_version: "encryptionkey@0"
I20260925 15:20:32.698530 8599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/ts-0/wal/instance:
uuid: "c170c1bf9b31476b9fcef7c83c46b0b6"
format_stamp: "Formatted at 2026-09-25 15:20:32 on dist-test-slave-4bk3"
server_key: "f7b8db00634d1997f7893fd191c83adf"
server_key_iv: "12c7c8b1b5a684fbd7fd58f2160c9bc4"
server_key_version: "encryptionkey@0"
I20260925 15:20:32.700100 8599 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:20:32.701370 8613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:32.701658 8599 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:20:32.701802 8599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/ts-0/wal
uuid: "c170c1bf9b31476b9fcef7c83c46b0b6"
format_stamp: "Formatted at 2026-09-25 15:20:32 on dist-test-slave-4bk3"
server_key: "f7b8db00634d1997f7893fd191c83adf"
server_key_iv: "12c7c8b1b5a684fbd7fd58f2160c9bc4"
server_key_version: "encryptionkey@0"
I20260925 15:20:32.701907 8599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:32.705631 8599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:32.705952 8599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:32.706108 8599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:32.706336 8599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:20:32.706859 8599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:20:32.706928 8599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:32.706980 8599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:20:32.707069 8599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:32.713276 8599 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.193:38423
I20260925 15:20:32.713500 8726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.193:38423 every 8 connection(s)
I20260925 15:20:32.713820 8599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:20:32.718235 8578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:32.715065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:53233 (local address 127.3.211.254:36785)
0925 15:20:32.715142 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:32.715144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:32.715503 (+ 359us) server_negotiation.cc:407] Connection header received
0925 15:20:32.715911 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:32.715913 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:32.715921 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:32.716006 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:32.717789 (+ 1783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:32.717791 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:32.717791 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:20:32.717797 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:32.717815 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:20:32.717820 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:32.717889 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:32.718127 (+ 238us) server_negotiation.cc:299] Negotiation successful
0925 15:20:32.718151 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 15:20:32.718357 8729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:32.714995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:36785 (local address 127.3.211.193:53233)
0925 15:20:32.715215 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:32.715259 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:32.715789 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:32.716214 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:32.716221 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:32.716235 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:32.717557 (+ 1322us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:20:32.717558 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:32.717596 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:20:32.717640 (+ 44us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:32.717952 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:32.717959 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:32.717963 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:20:32.718056 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 15:20:32.718131 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":107,"threads_started":1}
I20260925 15:20:32.719241 8727 heartbeater.cc:347] Connected to a master server at 127.3.211.254:36785
I20260925 15:20:32.719355 8727 heartbeater.cc:464] Registering TS with master...
I20260925 15:20:32.719545 8727 heartbeater.cc:511] Master 127.3.211.254:36785 requested a full tablet report, sending...
I20260925 15:20:32.720448 8539 ts_manager.cc:194] Registered new tserver with Master: c170c1bf9b31476b9fcef7c83c46b0b6 (127.3.211.193:38423)
I20260925 15:20:32.723784 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 8599
I20260925 15:20:32.724050 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/ts-0/wal/instance
I20260925 15:20:32.724617 3919 external_mini_cluster.cc:1428] Setting key dd92f12a496733bddda315fbbbe210f5
I20260925 15:20:32.725919 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.194:0
--local_ip_for_outbound_sockets=127.3.211.194
--webserver_interface=127.3.211.194
--webserver_port=0
--tserver_master_addrs=127.3.211.254:36785
--builtin_ntp_servers=127.3.211.212:39405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 15:20:32.811808 8730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:32.812041 8730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:32.812067 8730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:32.813416 8730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:32.813477 8730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:32.813508 8730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.194
I20260925 15:20:32.814875 8730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:39405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.211.194
--webserver_port=0
--tserver_master_addrs=127.3.211.254:36785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.194
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:32.815096 8730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:32.815368 8730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:32.820950 8735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:32.820950 8736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:32.821135 8730 server_base.cc:1034] running on GCE node
W20260925 15:20:32.820951 8738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:32.821440 8730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:32.821731 8730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:32.822965 8730 hybrid_clock.cc:648] HybridClock initialized: now 1790349632822905 us; error 73 us; skew 500 ppm
I20260925 15:20:32.824822 8730 webserver.cc:460] Webserver started at http://127.3.211.194:36869/ using document root <none> and password file <none>
I20260925 15:20:32.825202 8730 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:32.825289 8730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:32.825606 8730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:32.826859 8730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/ts-1/data/instance:
uuid: "56c49d8ff5ef4239bec372af2dd1bf75"
format_stamp: "Formatted at 2026-09-25 15:20:32 on dist-test-slave-4bk3"
server_key: "ef8d33e27f7d82987dd4aaffb3097073"
server_key_iv: "a163fb06548c27dd66111b1c2ed09e5d"
server_key_version: "encryptionkey@0"
I20260925 15:20:32.827260 8730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/ts-1/wal/instance:
uuid: "56c49d8ff5ef4239bec372af2dd1bf75"
format_stamp: "Formatted at 2026-09-25 15:20:32 on dist-test-slave-4bk3"
server_key: "ef8d33e27f7d82987dd4aaffb3097073"
server_key_iv: "a163fb06548c27dd66111b1c2ed09e5d"
server_key_version: "encryptionkey@0"
I20260925 15:20:32.828830 8730 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.003s
I20260925 15:20:32.829725 8744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:32.829926 8730 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:32.830039 8730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/ts-1/wal
uuid: "56c49d8ff5ef4239bec372af2dd1bf75"
format_stamp: "Formatted at 2026-09-25 15:20:32 on dist-test-slave-4bk3"
server_key: "ef8d33e27f7d82987dd4aaffb3097073"
server_key_iv: "a163fb06548c27dd66111b1c2ed09e5d"
server_key_version: "encryptionkey@0"
I20260925 15:20:32.830130 8730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:32.833856 8730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:32.834157 8730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:32.834308 8730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:32.834538 8730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:20:32.834883 8730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:20:32.834941 8730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:32.834985 8730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:20:32.835011 8730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:32.840889 8730 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.194:40881
I20260925 15:20:32.840969 8857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.194:40881 every 8 connection(s)
I20260925 15:20:32.841307 8730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
I20260925 15:20:32.844627 8578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:32.842104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:57397 (local address 127.3.211.254:36785)
0925 15:20:32.842168 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:32.842170 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:32.842499 (+ 329us) server_negotiation.cc:407] Connection header received
0925 15:20:32.842975 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:32.842976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:32.842981 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:32.843036 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:32.844267 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:32.844269 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:32.844269 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:20:32.844273 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:32.844287 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:20:32.844291 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:32.844436 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:32.844547 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 15:20:32.844560 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 15:20:32.844844 8860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:32.842093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:36785 (local address 127.3.211.194:57397)
0925 15:20:32.842368 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:32.842404 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:32.842880 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:32.843097 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:32.843100 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:32.843107 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:32.844082 (+ 975us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:20:32.844083 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:32.844114 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:20:32.844145 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:32.844448 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:32.844454 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:32.844460 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 15:20:32.844544 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 15:20:32.844600 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":92,"threads_started":1}
I20260925 15:20:32.845422 8858 heartbeater.cc:347] Connected to a master server at 127.3.211.254:36785
I20260925 15:20:32.845549 8858 heartbeater.cc:464] Registering TS with master...
I20260925 15:20:32.845719 8858 heartbeater.cc:511] Master 127.3.211.254:36785 requested a full tablet report, sending...
I20260925 15:20:32.846108 8539 ts_manager.cc:194] Registered new tserver with Master: 56c49d8ff5ef4239bec372af2dd1bf75 (127.3.211.194:40881)
I20260925 15:20:32.851287 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 8730
I20260925 15:20:32.851472 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/ts-1/wal/instance
I20260925 15:20:32.851694 3919 external_mini_cluster.cc:1428] Setting key c5a719c85557a8b257fe80d599235a59
I20260925 15:20:32.852793 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.195:0
--local_ip_for_outbound_sockets=127.3.211.195
--webserver_interface=127.3.211.195
--webserver_port=0
--tserver_master_addrs=127.3.211.254:36785
--builtin_ntp_servers=127.3.211.212:39405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 15:20:32.936484 8861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:32.936702 8861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:32.936726 8861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:32.937896 8861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:32.937935 8861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:32.937961 8861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.195
I20260925 15:20:32.939257 8861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:39405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.211.195
--webserver_port=0
--tserver_master_addrs=127.3.211.254:36785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.195
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:32.939472 8861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:32.939729 8861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:32.945336 8867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:32.945297 8869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:32.945307 8866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:32.945772 8861 server_base.cc:1034] running on GCE node
I20260925 15:20:32.946024 8861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:32.946321 8861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:32.947500 8861 hybrid_clock.cc:648] HybridClock initialized: now 1790349632947475 us; error 54 us; skew 500 ppm
I20260925 15:20:32.949136 8861 webserver.cc:460] Webserver started at http://127.3.211.195:38195/ using document root <none> and password file <none>
I20260925 15:20:32.949455 8861 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:32.949512 8861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:32.949677 8861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:32.951021 8861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/ts-2/data/instance:
uuid: "f262bcc3d765422fb96b2a5beca42956"
format_stamp: "Formatted at 2026-09-25 15:20:32 on dist-test-slave-4bk3"
server_key: "ae15376cc17b12ab38a9d75152820e4a"
server_key_iv: "fc8348537e5774c31c4b9f53649adbf4"
server_key_version: "encryptionkey@0"
I20260925 15:20:32.951455 8861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/ts-2/wal/instance:
uuid: "f262bcc3d765422fb96b2a5beca42956"
format_stamp: "Formatted at 2026-09-25 15:20:32 on dist-test-slave-4bk3"
server_key: "ae15376cc17b12ab38a9d75152820e4a"
server_key_iv: "fc8348537e5774c31c4b9f53649adbf4"
server_key_version: "encryptionkey@0"
I20260925 15:20:32.953042 8861 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260925 15:20:32.953912 8875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:32.954170 8861 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:32.954279 8861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/ts-2/wal
uuid: "f262bcc3d765422fb96b2a5beca42956"
format_stamp: "Formatted at 2026-09-25 15:20:32 on dist-test-slave-4bk3"
server_key: "ae15376cc17b12ab38a9d75152820e4a"
server_key_iv: "fc8348537e5774c31c4b9f53649adbf4"
server_key_version: "encryptionkey@0"
I20260925 15:20:32.954366 8861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:32.958029 8861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:32.958483 8861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:32.958685 8861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:32.958932 8861 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:20:32.959463 8861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:20:32.959522 8861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:32.959565 8861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:20:32.959599 8861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:32.965698 8861 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.195:42219
I20260925 15:20:32.965726 8988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.195:42219 every 8 connection(s)
I20260925 15:20:32.966100 8861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
I20260925 15:20:32.967854 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 8861
I20260925 15:20:32.967990 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/ts-2/wal/instance
I20260925 15:20:32.968219 3919 external_mini_cluster.cc:1428] Setting key 843f1d46eb5138811283fd7b78a82460
I20260925 15:20:32.969388 8578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:32.966923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:52609 (local address 127.3.211.254:36785)
0925 15:20:32.967052 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:32.967054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:32.967172 (+ 118us) server_negotiation.cc:407] Connection header received
0925 15:20:32.967691 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:32.967693 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:32.967699 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:32.967789 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:32.969116 (+ 1327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:32.969118 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:32.969118 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:20:32.969123 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:32.969138 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:20:32.969142 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:32.969202 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:32.969302 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 15:20:32.969318 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 15:20:32.969635 8991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:32.966850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:36785 (local address 127.3.211.195:52609)
0925 15:20:32.967039 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:32.967080 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:32.967546 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:32.967858 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:32.967861 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:32.967870 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:32.968947 (+ 1077us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:20:32.968947 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:32.968976 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:20:32.969006 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:32.969205 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:32.969209 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:32.969213 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:20:32.969312 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 15:20:32.969376 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":94,"threads_started":1}
I20260925 15:20:32.970175 8989 heartbeater.cc:347] Connected to a master server at 127.3.211.254:36785
I20260925 15:20:32.970263 8989 heartbeater.cc:464] Registering TS with master...
I20260925 15:20:32.970434 8989 heartbeater.cc:511] Master 127.3.211.254:36785 requested a full tablet report, sending...
I20260925 15:20:32.970891 8539 ts_manager.cc:194] Registered new tserver with Master: f262bcc3d765422fb96b2a5beca42956 (127.3.211.195:42219)
I20260925 15:20:32.980275 3919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:20:32.983204 8578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:32.981880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53184 (local address 127.3.211.254:36785)
0925 15:20:32.981967 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:32.981970 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:32.982067 (+ 97us) server_negotiation.cc:407] Connection header received
0925 15:20:32.982183 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:32.982185 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:32.982190 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:32.982255 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:32.982951 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:32.982952 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:32.982953 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:20:32.982957 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:32.982978 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:20:32.982982 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:32.983045 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:32.983104 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 15:20:32.983120 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 15:20:32.985304 8539 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53184:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:20:32.986939 8539 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:20:32.990936 8998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:32.989147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.194:40881 (local address 127.0.0.1:49202)
0925 15:20:32.989353 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:32.989402 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:32.989465 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:32.989757 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:32.989762 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:32.989767 (+ 5us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:32.990479 (+ 712us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:20:32.990479 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:32.990497 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:20:32.990523 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:32.990737 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:32.990744 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:32.990748 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:20:32.990835 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 15:20:32.990850 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":92,"threads_started":1}
I20260925 15:20:32.990957 8999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:32.989251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49202 (local address 127.3.211.194:40881)
0925 15:20:32.989588 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:32.989593 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:32.989603 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:20:32.989636 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:32.989639 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:32.989650 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:32.989761 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:32.990612 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:32.990618 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:32.990619 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:20:32.990623 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:32.990657 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:20:32.990676 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:32.990738 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:32.990845 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 15:20:32.990862 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":83,"threads_started":1}
I20260925 15:20:32.991201 8997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:32.989391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:38423 (local address 127.0.0.1:38210)
0925 15:20:32.989520 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:32.989532 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:32.989604 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:32.989826 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:32.989826 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:32.989828 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:32.990802 (+ 974us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:20:32.990803 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:32.990810 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:20:32.990823 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:32.991070 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:32.991070 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:32.991071 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 15:20:32.991114 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 15:20:32.991126 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":84,"threads_started":1}
I20260925 15:20:32.991277 9001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:32.989190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.195:42219 (local address 127.0.0.1:46198)
0925 15:20:32.989523 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:32.989533 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:32.989579 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:32.990418 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:32.990419 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:32.990422 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:32.990825 (+ 403us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:20:32.990826 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:32.990833 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:20:32.990844 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:32.991071 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:32.991072 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:32.991073 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 15:20:32.991124 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 15:20:32.991138 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":88,"threads_started":1}
I20260925 15:20:32.991320 9000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:32.989299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46198 (local address 127.3.211.195:42219)
0925 15:20:32.990264 (+ 965us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:32.990267 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:32.990276 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:20:32.990322 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:32.990323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:32.990330 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:32.990389 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:32.990922 (+ 533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:32.990929 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:32.990929 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:20:32.990933 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:32.990963 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:20:32.990989 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:32.991043 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:32.991231 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 15:20:32.991246 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":926,"thread_start_us":92,"threads_started":1}
I20260925 15:20:32.991470 9002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:32.989498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38210 (local address 127.3.211.193:38423)
0925 15:20:32.989683 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:32.989687 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:32.989696 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:20:32.989720 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:32.989722 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:32.989731 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:32.989808 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:32.990911 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:32.990917 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:32.990917 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:20:32.990921 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:32.990953 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:20:32.990973 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:32.991010 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:32.991365 (+ 355us) server_negotiation.cc:299] Negotiation successful
0925 15:20:32.991381 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":82,"threads_started":1}
I20260925 15:20:32.992185 8923 tablet_service.cc:1467] Processing CreateTablet for tablet 45177b836302495ab0dda3ced2e10fb4 (DEFAULT_TABLE table=test-table [id=908f6b7361ff450cbc06bcc2056c1c1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:20:32.992434 8792 tablet_service.cc:1467] Processing CreateTablet for tablet 45177b836302495ab0dda3ced2e10fb4 (DEFAULT_TABLE table=test-table [id=908f6b7361ff450cbc06bcc2056c1c1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:20:32.992580 8923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45177b836302495ab0dda3ced2e10fb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:32.992774 8792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45177b836302495ab0dda3ced2e10fb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:32.995322 9003 tablet_bootstrap.cc:492] T 45177b836302495ab0dda3ced2e10fb4 P f262bcc3d765422fb96b2a5beca42956: Bootstrap starting.
I20260925 15:20:32.995716 9004 tablet_bootstrap.cc:492] T 45177b836302495ab0dda3ced2e10fb4 P 56c49d8ff5ef4239bec372af2dd1bf75: Bootstrap starting.
I20260925 15:20:32.996001 9003 tablet_bootstrap.cc:654] T 45177b836302495ab0dda3ced2e10fb4 P f262bcc3d765422fb96b2a5beca42956: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:32.995878 8661 tablet_service.cc:1467] Processing CreateTablet for tablet 45177b836302495ab0dda3ced2e10fb4 (DEFAULT_TABLE table=test-table [id=908f6b7361ff450cbc06bcc2056c1c1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:20:32.996232 8661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45177b836302495ab0dda3ced2e10fb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:32.996359 9003 log.cc:824] T 45177b836302495ab0dda3ced2e10fb4 P f262bcc3d765422fb96b2a5beca42956: Log is configured to *not* fsync() on all Append() calls
I20260925 15:20:32.996392 9004 tablet_bootstrap.cc:654] T 45177b836302495ab0dda3ced2e10fb4 P 56c49d8ff5ef4239bec372af2dd1bf75: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:32.996717 9004 log.cc:824] T 45177b836302495ab0dda3ced2e10fb4 P 56c49d8ff5ef4239bec372af2dd1bf75: Log is configured to *not* fsync() on all Append() calls
I20260925 15:20:32.997201 9003 tablet_bootstrap.cc:492] T 45177b836302495ab0dda3ced2e10fb4 P f262bcc3d765422fb96b2a5beca42956: No bootstrap required, opened a new log
I20260925 15:20:32.997275 9003 ts_tablet_manager.cc:1397] T 45177b836302495ab0dda3ced2e10fb4 P f262bcc3d765422fb96b2a5beca42956: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:20:32.997478 9004 tablet_bootstrap.cc:492] T 45177b836302495ab0dda3ced2e10fb4 P 56c49d8ff5ef4239bec372af2dd1bf75: No bootstrap required, opened a new log
I20260925 15:20:32.997545 9004 ts_tablet_manager.cc:1397] T 45177b836302495ab0dda3ced2e10fb4 P 56c49d8ff5ef4239bec372af2dd1bf75: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 15:20:32.999745 9004 raft_consensus.cc:348] T 45177b836302495ab0dda3ced2e10fb4 P 56c49d8ff5ef4239bec372af2dd1bf75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56c49d8ff5ef4239bec372af2dd1bf75" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40881 } } peers { permanent_uuid: "c170c1bf9b31476b9fcef7c83c46b0b6" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38423 } } peers { permanent_uuid: "f262bcc3d765422fb96b2a5beca42956" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 42219 } }
I20260925 15:20:32.999980 9004 raft_consensus.cc:374] T 45177b836302495ab0dda3ced2e10fb4 P 56c49d8ff5ef4239bec372af2dd1bf75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:33.000026 9004 raft_consensus.cc:729] T 45177b836302495ab0dda3ced2e10fb4 P 56c49d8ff5ef4239bec372af2dd1bf75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56c49d8ff5ef4239bec372af2dd1bf75, State: Initialized, Role: FOLLOWER
I20260925 15:20:32.999945 9007 tablet_bootstrap.cc:492] T 45177b836302495ab0dda3ced2e10fb4 P c170c1bf9b31476b9fcef7c83c46b0b6: Bootstrap starting.
I20260925 15:20:33.000173 9004 consensus_queue.cc:260] T 45177b836302495ab0dda3ced2e10fb4 P 56c49d8ff5ef4239bec372af2dd1bf75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56c49d8ff5ef4239bec372af2dd1bf75" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40881 } } peers { permanent_uuid: "c170c1bf9b31476b9fcef7c83c46b0b6" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38423 } } peers { permanent_uuid: "f262bcc3d765422fb96b2a5beca42956" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 42219 } }
I20260925 15:20:33.000093 9003 raft_consensus.cc:348] T 45177b836302495ab0dda3ced2e10fb4 P f262bcc3d765422fb96b2a5beca42956 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56c49d8ff5ef4239bec372af2dd1bf75" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40881 } } peers { permanent_uuid: "c170c1bf9b31476b9fcef7c83c46b0b6" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38423 } } peers { permanent_uuid: "f262bcc3d765422fb96b2a5beca42956" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 42219 } }
I20260925 15:20:33.000262 9003 raft_consensus.cc:374] T 45177b836302495ab0dda3ced2e10fb4 P f262bcc3d765422fb96b2a5beca42956 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:33.000308 9003 raft_consensus.cc:729] T 45177b836302495ab0dda3ced2e10fb4 P f262bcc3d765422fb96b2a5beca42956 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f262bcc3d765422fb96b2a5beca42956, State: Initialized, Role: FOLLOWER
I20260925 15:20:33.000478 9004 ts_tablet_manager.cc:1428] T 45177b836302495ab0dda3ced2e10fb4 P 56c49d8ff5ef4239bec372af2dd1bf75: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 15:20:33.000464 9003 consensus_queue.cc:260] T 45177b836302495ab0dda3ced2e10fb4 P f262bcc3d765422fb96b2a5beca42956 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56c49d8ff5ef4239bec372af2dd1bf75" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40881 } } peers { permanent_uuid: "c170c1bf9b31476b9fcef7c83c46b0b6" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38423 } } peers { permanent_uuid: "f262bcc3d765422fb96b2a5beca42956" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 42219 } }
I20260925 15:20:33.000627 8858 heartbeater.cc:503] Master 127.3.211.254:36785 was elected leader, sending a full tablet report...
I20260925 15:20:33.000763 9003 ts_tablet_manager.cc:1428] T 45177b836302495ab0dda3ced2e10fb4 P f262bcc3d765422fb96b2a5beca42956: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260925 15:20:33.000892 8989 heartbeater.cc:503] Master 127.3.211.254:36785 was elected leader, sending a full tablet report...
I20260925 15:20:33.001061 9007 tablet_bootstrap.cc:654] T 45177b836302495ab0dda3ced2e10fb4 P c170c1bf9b31476b9fcef7c83c46b0b6: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:33.001425 9007 log.cc:824] T 45177b836302495ab0dda3ced2e10fb4 P c170c1bf9b31476b9fcef7c83c46b0b6: Log is configured to *not* fsync() on all Append() calls
I20260925 15:20:33.002230 9007 tablet_bootstrap.cc:492] T 45177b836302495ab0dda3ced2e10fb4 P c170c1bf9b31476b9fcef7c83c46b0b6: No bootstrap required, opened a new log
I20260925 15:20:33.002298 9007 ts_tablet_manager.cc:1397] T 45177b836302495ab0dda3ced2e10fb4 P c170c1bf9b31476b9fcef7c83c46b0b6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:20:33.004133 9007 raft_consensus.cc:348] T 45177b836302495ab0dda3ced2e10fb4 P c170c1bf9b31476b9fcef7c83c46b0b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56c49d8ff5ef4239bec372af2dd1bf75" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40881 } } peers { permanent_uuid: "c170c1bf9b31476b9fcef7c83c46b0b6" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38423 } } peers { permanent_uuid: "f262bcc3d765422fb96b2a5beca42956" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 42219 } }
I20260925 15:20:33.004365 9007 raft_consensus.cc:374] T 45177b836302495ab0dda3ced2e10fb4 P c170c1bf9b31476b9fcef7c83c46b0b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:33.004452 9007 raft_consensus.cc:729] T 45177b836302495ab0dda3ced2e10fb4 P c170c1bf9b31476b9fcef7c83c46b0b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c170c1bf9b31476b9fcef7c83c46b0b6, State: Initialized, Role: FOLLOWER
I20260925 15:20:33.004588 9007 consensus_queue.cc:260] T 45177b836302495ab0dda3ced2e10fb4 P c170c1bf9b31476b9fcef7c83c46b0b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56c49d8ff5ef4239bec372af2dd1bf75" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40881 } } peers { permanent_uuid: "c170c1bf9b31476b9fcef7c83c46b0b6" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38423 } } peers { permanent_uuid: "f262bcc3d765422fb96b2a5beca42956" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 42219 } }
I20260925 15:20:33.004866 9007 ts_tablet_manager.cc:1428] T 45177b836302495ab0dda3ced2e10fb4 P c170c1bf9b31476b9fcef7c83c46b0b6: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260925 15:20:33.004917 8727 heartbeater.cc:503] Master 127.3.211.254:36785 was elected leader, sending a full tablet report...
W20260925 15:20:33.091938 8859 tablet.cc:2369] T 45177b836302495ab0dda3ced2e10fb4 P 56c49d8ff5ef4239bec372af2dd1bf75: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:20:33.214664 8728 tablet.cc:2369] T 45177b836302495ab0dda3ced2e10fb4 P c170c1bf9b31476b9fcef7c83c46b0b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:20:33.216673 8990 tablet.cc:2369] T 45177b836302495ab0dda3ced2e10fb4 P f262bcc3d765422fb96b2a5beca42956: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:20:33.309336 9008 raft_consensus.cc:482] T 45177b836302495ab0dda3ced2e10fb4 P 56c49d8ff5ef4239bec372af2dd1bf75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:20:33.309477 9008 raft_consensus.cc:504] T 45177b836302495ab0dda3ced2e10fb4 P 56c49d8ff5ef4239bec372af2dd1bf75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56c49d8ff5ef4239bec372af2dd1bf75" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40881 } } peers { permanent_uuid: "c170c1bf9b31476b9fcef7c83c46b0b6" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38423 } } peers { permanent_uuid: "f262bcc3d765422fb96b2a5beca42956" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 42219 } }
I20260925 15:20:33.309985 9008 leader_election.cc:288] T 45177b836302495ab0dda3ced2e10fb4 P 56c49d8ff5ef4239bec372af2dd1bf75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c170c1bf9b31476b9fcef7c83c46b0b6 (127.3.211.193:38423), f262bcc3d765422fb96b2a5beca42956 (127.3.211.195:42219)
I20260925 15:20:33.315263 9002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:33.310262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:35243 (local address 127.3.211.193:38423)
0925 15:20:33.310386 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:33.310388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:33.310397 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:20:33.310497 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:33.310498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:33.310504 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:33.310570 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:33.314907 (+ 4337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:33.314909 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:33.314910 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:20:33.314922 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:33.314941 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:20:33.314947 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:33.315024 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:33.315162 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 15:20:33.315179 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260925 15:20:33.315297 8860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:33.310235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:38423 (local address 127.3.211.194:35243)
0925 15:20:33.310325 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:33.310341 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:33.310396 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:33.310646 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:33.310647 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:33.310650 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:33.314725 (+ 4075us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:20:33.314726 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:33.314738 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:20:33.314766 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:33.315087 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:33.315088 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:33.315089 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 15:20:33.315154 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 15:20:33.315173 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 15:20:33.315330 9012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:33.310235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.195:42219 (local address 127.3.211.194:57333)
0925 15:20:33.310510 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:33.310521 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:33.310573 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:33.310741 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:33.310742 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:33.310744 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:33.314789 (+ 4045us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:20:33.314790 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:33.314798 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:20:33.314852 (+ 54us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:33.315162 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:33.315163 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:33.315163 (+ 0us) client_negotiation.cc:769] Sending connection context
0925 15:20:33.315219 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 15:20:33.315234 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":88,"thread_start_us":111,"threads_started":1}
I20260925 15:20:33.315340 9000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:33.310280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:57333 (local address 127.3.211.195:42219)
0925 15:20:33.310354 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:33.310356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:33.310622 (+ 266us) server_negotiation.cc:407] Connection header received
0925 15:20:33.310650 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:33.310651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:33.310656 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:33.310728 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:33.315022 (+ 4294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:33.315025 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:33.315025 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:20:33.315033 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:33.315056 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:20:33.315062 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:33.315145 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:33.315238 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 15:20:33.315256 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 15:20:33.315730 8681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45177b836302495ab0dda3ced2e10fb4" candidate_uuid: "56c49d8ff5ef4239bec372af2dd1bf75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c170c1bf9b31476b9fcef7c83c46b0b6" is_pre_election: true
I20260925 15:20:33.315775 8943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45177b836302495ab0dda3ced2e10fb4" candidate_uuid: "56c49d8ff5ef4239bec372af2dd1bf75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f262bcc3d765422fb96b2a5beca42956" is_pre_election: true
I20260925 15:20:33.315944 8681 raft_consensus.cc:2454] T 45177b836302495ab0dda3ced2e10fb4 P c170c1bf9b31476b9fcef7c83c46b0b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56c49d8ff5ef4239bec372af2dd1bf75 in term 0.
I20260925 15:20:33.315960 8943 raft_consensus.cc:2454] T 45177b836302495ab0dda3ced2e10fb4 P f262bcc3d765422fb96b2a5beca42956 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 56c49d8ff5ef4239bec372af2dd1bf75 in term 0.
I20260925 15:20:33.316238 8746 leader_election.cc:302] T 45177b836302495ab0dda3ced2e10fb4 P 56c49d8ff5ef4239bec372af2dd1bf75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56c49d8ff5ef4239bec372af2dd1bf75, c170c1bf9b31476b9fcef7c83c46b0b6; no voters:
I20260925 15:20:33.316421 9008 raft_consensus.cc:2781] T 45177b836302495ab0dda3ced2e10fb4 P 56c49d8ff5ef4239bec372af2dd1bf75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:20:33.316484 9008 raft_consensus.cc:482] T 45177b836302495ab0dda3ced2e10fb4 P 56c49d8ff5ef4239bec372af2dd1bf75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:20:33.316519 9008 raft_consensus.cc:3037] T 45177b836302495ab0dda3ced2e10fb4 P 56c49d8ff5ef4239bec372af2dd1bf75 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:33.317425 9008 raft_consensus.cc:504] T 45177b836302495ab0dda3ced2e10fb4 P 56c49d8ff5ef4239bec372af2dd1bf75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56c49d8ff5ef4239bec372af2dd1bf75" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40881 } } peers { permanent_uuid: "c170c1bf9b31476b9fcef7c83c46b0b6" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38423 } } peers { permanent_uuid: "f262bcc3d765422fb96b2a5beca42956" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 42219 } }
I20260925 15:20:33.317662 9008 leader_election.cc:288] T 45177b836302495ab0dda3ced2e10fb4 P 56c49d8ff5ef4239bec372af2dd1bf75 [CANDIDATE]: Term 1 election: Requested vote from peers c170c1bf9b31476b9fcef7c83c46b0b6 (127.3.211.193:38423), f262bcc3d765422fb96b2a5beca42956 (127.3.211.195:42219)
I20260925 15:20:33.317776 8681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45177b836302495ab0dda3ced2e10fb4" candidate_uuid: "56c49d8ff5ef4239bec372af2dd1bf75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c170c1bf9b31476b9fcef7c83c46b0b6"
I20260925 15:20:33.317788 8943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45177b836302495ab0dda3ced2e10fb4" candidate_uuid: "56c49d8ff5ef4239bec372af2dd1bf75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f262bcc3d765422fb96b2a5beca42956"
I20260925 15:20:33.317865 8943 raft_consensus.cc:3037] T 45177b836302495ab0dda3ced2e10fb4 P f262bcc3d765422fb96b2a5beca42956 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:33.317881 8681 raft_consensus.cc:3037] T 45177b836302495ab0dda3ced2e10fb4 P c170c1bf9b31476b9fcef7c83c46b0b6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:33.318588 8943 raft_consensus.cc:2454] T 45177b836302495ab0dda3ced2e10fb4 P f262bcc3d765422fb96b2a5beca42956 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56c49d8ff5ef4239bec372af2dd1bf75 in term 1.
I20260925 15:20:33.318594 8681 raft_consensus.cc:2454] T 45177b836302495ab0dda3ced2e10fb4 P c170c1bf9b31476b9fcef7c83c46b0b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 56c49d8ff5ef4239bec372af2dd1bf75 in term 1.
I20260925 15:20:33.318775 8746 leader_election.cc:302] T 45177b836302495ab0dda3ced2e10fb4 P 56c49d8ff5ef4239bec372af2dd1bf75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56c49d8ff5ef4239bec372af2dd1bf75, c170c1bf9b31476b9fcef7c83c46b0b6; no voters:
I20260925 15:20:33.318873 9008 raft_consensus.cc:2781] T 45177b836302495ab0dda3ced2e10fb4 P 56c49d8ff5ef4239bec372af2dd1bf75 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:20:33.319042 9008 raft_consensus.cc:686] T 45177b836302495ab0dda3ced2e10fb4 P 56c49d8ff5ef4239bec372af2dd1bf75 [term 1 LEADER]: Becoming Leader. State: Replica: 56c49d8ff5ef4239bec372af2dd1bf75, State: Running, Role: LEADER
I20260925 15:20:33.319150 9008 consensus_queue.cc:237] T 45177b836302495ab0dda3ced2e10fb4 P 56c49d8ff5ef4239bec372af2dd1bf75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56c49d8ff5ef4239bec372af2dd1bf75" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40881 } } peers { permanent_uuid: "c170c1bf9b31476b9fcef7c83c46b0b6" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38423 } } peers { permanent_uuid: "f262bcc3d765422fb96b2a5beca42956" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 42219 } }
I20260925 15:20:33.320091 8539 catalog_manager.cc:5495] T 45177b836302495ab0dda3ced2e10fb4 P 56c49d8ff5ef4239bec372af2dd1bf75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 56c49d8ff5ef4239bec372af2dd1bf75 (127.3.211.194). New cstate: current_term: 1 leader_uuid: "56c49d8ff5ef4239bec372af2dd1bf75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56c49d8ff5ef4239bec372af2dd1bf75" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c170c1bf9b31476b9fcef7c83c46b0b6" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f262bcc3d765422fb96b2a5beca42956" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 42219 } health_report { overall_health: UNKNOWN } } }
I20260925 15:20:33.347663 8999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:33.346344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49208 (local address 127.3.211.194:40881)
0925 15:20:33.346403 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:33.346405 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:33.346430 (+ 25us) server_negotiation.cc:407] Connection header received
0925 15:20:33.346472 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:33.346473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:33.346488 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:33.346551 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:33.347282 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:33.347283 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:33.347283 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:20:33.347288 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:33.347304 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:20:33.347308 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:33.347449 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:33.347552 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 15:20:33.347570 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 15:20:33.349581 8943 raft_consensus.cc:1261] T 45177b836302495ab0dda3ced2e10fb4 P f262bcc3d765422fb96b2a5beca42956 [term 1 FOLLOWER]: Refusing update from remote peer 56c49d8ff5ef4239bec372af2dd1bf75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:20:33.349655 8681 raft_consensus.cc:1261] T 45177b836302495ab0dda3ced2e10fb4 P c170c1bf9b31476b9fcef7c83c46b0b6 [term 1 FOLLOWER]: Refusing update from remote peer 56c49d8ff5ef4239bec372af2dd1bf75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:20:33.349951 9013 consensus_queue.cc:1035] T 45177b836302495ab0dda3ced2e10fb4 P 56c49d8ff5ef4239bec372af2dd1bf75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f262bcc3d765422fb96b2a5beca42956" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 42219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:20:33.350109 9008 consensus_queue.cc:1035] T 45177b836302495ab0dda3ced2e10fb4 P 56c49d8ff5ef4239bec372af2dd1bf75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c170c1bf9b31476b9fcef7c83c46b0b6" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:20:33.352450 9016 mvcc.cc:205] Tried to move back new op lower bound from 7333272098197110784 to 7333272098075684864. Current Snapshot: MvccSnapshot[applied={T|T < 7333272098197110784}]
I20260925 15:20:33.355032 8538 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:20:33.355150 8538 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:20:33.355973 8538 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 026e6f1ff967497eb3d8d2bedad98388: Sending DeleteTablet for 3 replicas of tablet 45177b836302495ab0dda3ced2e10fb4
I20260925 15:20:33.356388 8792 tablet_service.cc:1514] Processing DeleteTablet for tablet 45177b836302495ab0dda3ced2e10fb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:20:33 UTC) from {username='slave'} at 127.0.0.1:49202
I20260925 15:20:33.356422 8661 tablet_service.cc:1514] Processing DeleteTablet for tablet 45177b836302495ab0dda3ced2e10fb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:20:33 UTC) from {username='slave'} at 127.0.0.1:38210
I20260925 15:20:33.356446 8923 tablet_service.cc:1514] Processing DeleteTablet for tablet 45177b836302495ab0dda3ced2e10fb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:20:33 UTC) from {username='slave'} at 127.0.0.1:46198
I20260925 15:20:33.356633 9026 tablet_replica.cc:330] T 45177b836302495ab0dda3ced2e10fb4 P c170c1bf9b31476b9fcef7c83c46b0b6: stopping tablet replica
I20260925 15:20:33.356726 9026 raft_consensus.cc:2229] T 45177b836302495ab0dda3ced2e10fb4 P c170c1bf9b31476b9fcef7c83c46b0b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:20:33.356767 9025 tablet_replica.cc:330] T 45177b836302495ab0dda3ced2e10fb4 P 56c49d8ff5ef4239bec372af2dd1bf75: stopping tablet replica
I20260925 15:20:33.356828 9026 raft_consensus.cc:2258] T 45177b836302495ab0dda3ced2e10fb4 P c170c1bf9b31476b9fcef7c83c46b0b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:20:33.356904 9025 raft_consensus.cc:2229] T 45177b836302495ab0dda3ced2e10fb4 P 56c49d8ff5ef4239bec372af2dd1bf75 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:20:33.356921 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 8599
I20260925 15:20:33.357007 9025 raft_consensus.cc:2258] T 45177b836302495ab0dda3ced2e10fb4 P 56c49d8ff5ef4239bec372af2dd1bf75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:20:33.358043 9025 ts_tablet_manager.cc:1905] T 45177b836302495ab0dda3ced2e10fb4 P 56c49d8ff5ef4239bec372af2dd1bf75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:20:33.359369 9027 tablet_replica.cc:330] T 45177b836302495ab0dda3ced2e10fb4 P f262bcc3d765422fb96b2a5beca42956: stopping tablet replica
I20260925 15:20:33.359539 9027 raft_consensus.cc:2229] T 45177b836302495ab0dda3ced2e10fb4 P f262bcc3d765422fb96b2a5beca42956 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:20:33.359706 9027 raft_consensus.cc:2258] T 45177b836302495ab0dda3ced2e10fb4 P f262bcc3d765422fb96b2a5beca42956 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:20:33.359999 9025 ts_tablet_manager.cc:1918] T 45177b836302495ab0dda3ced2e10fb4 P 56c49d8ff5ef4239bec372af2dd1bf75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:20:33.360085 9025 log.cc:1196] T 45177b836302495ab0dda3ced2e10fb4 P 56c49d8ff5ef4239bec372af2dd1bf75: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/ts-1/wal/wals/45177b836302495ab0dda3ced2e10fb4
I20260925 15:20:33.360401 9027 ts_tablet_manager.cc:1905] T 45177b836302495ab0dda3ced2e10fb4 P f262bcc3d765422fb96b2a5beca42956: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:20:33.360426 9025 ts_tablet_manager.cc:1939] T 45177b836302495ab0dda3ced2e10fb4 P 56c49d8ff5ef4239bec372af2dd1bf75: Deleting consensus metadata
I20260925 15:20:33.360850 8524 catalog_manager.cc:4841] TS 56c49d8ff5ef4239bec372af2dd1bf75 (127.3.211.194:40881): tablet 45177b836302495ab0dda3ced2e10fb4 (table test-table [id=908f6b7361ff450cbc06bcc2056c1c1b]) successfully deleted
I20260925 15:20:33.361943 9027 ts_tablet_manager.cc:1918] T 45177b836302495ab0dda3ced2e10fb4 P f262bcc3d765422fb96b2a5beca42956: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:20:33.362043 9027 log.cc:1196] T 45177b836302495ab0dda3ced2e10fb4 P f262bcc3d765422fb96b2a5beca42956: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790349610595785-3919-0/minicluster-data/ts-2/wal/wals/45177b836302495ab0dda3ced2e10fb4
I20260925 15:20:33.362317 9027 ts_tablet_manager.cc:1939] T 45177b836302495ab0dda3ced2e10fb4 P f262bcc3d765422fb96b2a5beca42956: Deleting consensus metadata
I20260925 15:20:33.362762 8523 catalog_manager.cc:4841] TS f262bcc3d765422fb96b2a5beca42956 (127.3.211.195:42219): tablet 45177b836302495ab0dda3ced2e10fb4 (table test-table [id=908f6b7361ff450cbc06bcc2056c1c1b]) successfully deleted
W20260925 15:20:33.363932 8524 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.211.193:38423 (error 108)
I20260925 15:20:33.364255 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 8730
I20260925 15:20:33.364909 8997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:33.364373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:38423 (local address 127.0.0.1:38226)
0925 15:20:33.364826 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:33.364872 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.193:38423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433}
W20260925 15:20:33.365057 8524 catalog_manager.cc:4584] TS c170c1bf9b31476b9fcef7c83c46b0b6 (127.3.211.193:38423): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 45177b836302495ab0dda3ced2e10fb4: Network error: Client connection negotiation failed: client connection to 127.3.211.193:38423: connect: Connection refused (error 111)
I20260925 15:20:33.370689 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 8861
I20260925 15:20:33.375463 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 8508
2026-09-25T15:20:33Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (981 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 15:20:33.396987 3919 mini_oidc.cc:152] Starting JWKS server
I20260925 15:20:33.398422 3919 webserver.cc:460] Webserver started at https://127.0.0.1:36979/ using document root <none> and password file <none>
I20260925 15:20:33.398516 3919 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 15:20:33.398744 3919 webserver.cc:460] Webserver started at http://127.0.0.1:44645/ using document root <none> and password file <none>
2026-09-25T15:20:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:20:33Z Disabled control of system clock
I20260925 15:20:33.407649 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:41095
--webserver_interface=127.3.211.254
--webserver_port=0
--builtin_ntp_servers=127.3.211.212:42111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.211.254:41095
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36979/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349610595785-3919-0/testchainca.cert with env {}
W20260925 15:20:33.491535 9034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:33.491768 9034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:33.491793 9034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:33.493166 9034 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:20:33.493232 9034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:33.493245 9034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:33.493258 9034 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:20:33.493271 9034 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 15:20:33.493284 9034 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 15:20:33.493296 9034 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36979/jwks.json
I20260925 15:20:33.495072 9034 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:42111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349610595785-3919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.211.254:41095
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:41095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36979/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.211.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349610595785-3919-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:33.495378 9034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:33.495647 9034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:33.501001 9040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:33.501014 9039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:33.501416 9042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:33.501356 9034 server_base.cc:1034] running on GCE node
I20260925 15:20:33.501677 9034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:33.501964 9034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:33.503166 9034 hybrid_clock.cc:648] HybridClock initialized: now 1790349633503166 us; error 56 us; skew 500 ppm
I20260925 15:20:33.504637 9034 webserver.cc:460] Webserver started at http://127.3.211.254:44299/ using document root <none> and password file <none>
I20260925 15:20:33.504917 9034 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:33.504993 9034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:33.505151 9034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:33.506278 9034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349610595785-3919-0/minicluster-data/master-0/data/instance:
uuid: "fdd23254edbf48b6a87e181331b59159"
format_stamp: "Formatted at 2026-09-25 15:20:33 on dist-test-slave-4bk3"
server_key: "622263f4f650bde80c9159cec3b775db"
server_key_iv: "e1245a39f5da95bac407274b96cc1ced"
server_key_version: "encryptionkey@0"
I20260925 15:20:33.506676 9034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349610595785-3919-0/minicluster-data/master-0/wal/instance:
uuid: "fdd23254edbf48b6a87e181331b59159"
format_stamp: "Formatted at 2026-09-25 15:20:33 on dist-test-slave-4bk3"
server_key: "622263f4f650bde80c9159cec3b775db"
server_key_iv: "e1245a39f5da95bac407274b96cc1ced"
server_key_version: "encryptionkey@0"
I20260925 15:20:33.508065 9034 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:20:33.508998 9048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:33.509229 9034 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:33.509346 9034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349610595785-3919-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349610595785-3919-0/minicluster-data/master-0/wal
uuid: "fdd23254edbf48b6a87e181331b59159"
format_stamp: "Formatted at 2026-09-25 15:20:33 on dist-test-slave-4bk3"
server_key: "622263f4f650bde80c9159cec3b775db"
server_key_iv: "e1245a39f5da95bac407274b96cc1ced"
server_key_version: "encryptionkey@0"
I20260925 15:20:33.509441 9034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349610595785-3919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349610595785-3919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349610595785-3919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:33.524730 9034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:33.525066 9034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:33.525223 9034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:33.529237 9034 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.254:41095
I20260925 15:20:33.529295 9100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.254:41095 every 8 connection(s)
I20260925 15:20:33.529657 9034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
I20260925 15:20:33.530663 9101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:33.533115 9101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdd23254edbf48b6a87e181331b59159: Bootstrap starting.
I20260925 15:20:33.533139 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 9034
I20260925 15:20:33.533288 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349610595785-3919-0/minicluster-data/master-0/wal/instance
I20260925 15:20:33.533485 3919 external_mini_cluster.cc:1428] Setting key 480849dedc7a97c226bb73e4e99d5ff1
I20260925 15:20:33.533914 9101 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fdd23254edbf48b6a87e181331b59159: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:33.534312 9101 log.cc:824] T 00000000000000000000000000000000 P fdd23254edbf48b6a87e181331b59159: Log is configured to *not* fsync() on all Append() calls
I20260925 15:20:33.535105 9101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdd23254edbf48b6a87e181331b59159: No bootstrap required, opened a new log
I20260925 15:20:33.537516 9101 raft_consensus.cc:348] T 00000000000000000000000000000000 P fdd23254edbf48b6a87e181331b59159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdd23254edbf48b6a87e181331b59159" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 41095 } }
I20260925 15:20:33.537667 9101 raft_consensus.cc:374] T 00000000000000000000000000000000 P fdd23254edbf48b6a87e181331b59159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:33.537690 9101 raft_consensus.cc:729] T 00000000000000000000000000000000 P fdd23254edbf48b6a87e181331b59159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdd23254edbf48b6a87e181331b59159, State: Initialized, Role: FOLLOWER
I20260925 15:20:33.537854 9101 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdd23254edbf48b6a87e181331b59159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdd23254edbf48b6a87e181331b59159" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 41095 } }
I20260925 15:20:33.537930 9101 raft_consensus.cc:388] T 00000000000000000000000000000000 P fdd23254edbf48b6a87e181331b59159 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:20:33.537958 9101 raft_consensus.cc:482] T 00000000000000000000000000000000 P fdd23254edbf48b6a87e181331b59159 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:20:33.537994 9101 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fdd23254edbf48b6a87e181331b59159 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:33.538766 9101 raft_consensus.cc:504] T 00000000000000000000000000000000 P fdd23254edbf48b6a87e181331b59159 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdd23254edbf48b6a87e181331b59159" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 41095 } }
I20260925 15:20:33.538893 9101 leader_election.cc:302] T 00000000000000000000000000000000 P fdd23254edbf48b6a87e181331b59159 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fdd23254edbf48b6a87e181331b59159; no voters:
I20260925 15:20:33.539063 9101 leader_election.cc:288] T 00000000000000000000000000000000 P fdd23254edbf48b6a87e181331b59159 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:20:33.539137 9106 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fdd23254edbf48b6a87e181331b59159 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:20:33.539288 9106 raft_consensus.cc:686] T 00000000000000000000000000000000 P fdd23254edbf48b6a87e181331b59159 [term 1 LEADER]: Becoming Leader. State: Replica: fdd23254edbf48b6a87e181331b59159, State: Running, Role: LEADER
I20260925 15:20:33.539366 9101 sys_catalog.cc:564] T 00000000000000000000000000000000 P fdd23254edbf48b6a87e181331b59159 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:20:33.539384 9106 consensus_queue.cc:237] T 00000000000000000000000000000000 P fdd23254edbf48b6a87e181331b59159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdd23254edbf48b6a87e181331b59159" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 41095 } }
I20260925 15:20:33.539768 9106 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdd23254edbf48b6a87e181331b59159 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdd23254edbf48b6a87e181331b59159. Latest consensus state: current_term: 1 leader_uuid: "fdd23254edbf48b6a87e181331b59159" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdd23254edbf48b6a87e181331b59159" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 41095 } } }
I20260925 15:20:33.539845 9106 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdd23254edbf48b6a87e181331b59159 [sys.catalog]: This master's current role is: LEADER
I20260925 15:20:33.540009 9107 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdd23254edbf48b6a87e181331b59159 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fdd23254edbf48b6a87e181331b59159" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdd23254edbf48b6a87e181331b59159" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 41095 } } }
I20260925 15:20:33.540087 9107 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdd23254edbf48b6a87e181331b59159 [sys.catalog]: This master's current role is: LEADER
I20260925 15:20:33.540398 9112 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:20:33.540715 9104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:33.534029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59664 (local address 127.3.211.254:41095)
0925 15:20:33.534278 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:33.534287 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:33.534322 (+ 35us) server_negotiation.cc:407] Connection header received
0925 15:20:33.534932 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:33.534941 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:33.535017 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:33.535191 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:33.535486 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:33.536909 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:33.537759 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:33.538369 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:33.538499 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:33.539402 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:33.539406 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:33.539407 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:20:33.539735 (+ 328us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:33.539778 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:20:33.539793 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:33.540276 (+ 483us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:33.540313 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 15:20:33.540430 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":87,"threads_started":1}
I20260925 15:20:33.540946 9112 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:20:33.542331 3919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:20:33.543123 9112 catalog_manager.cc:1347] Generated new cluster ID: a5f701c3575b4ed69dbab6d6c7971d33
I20260925 15:20:33.543190 9112 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:20:33.564908 9112 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:20:33.565528 9112 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:20:33.573587 9112 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fdd23254edbf48b6a87e181331b59159: Generated new TSK 0
I20260925 15:20:33.573843 9112 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 15:20:33.604570 9131 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
I20260925 15:20:33.604681 9104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:33.588150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59672 (local address 127.3.211.254:41095)
0925 15:20:33.588219 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:33.588222 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:33.588296 (+ 74us) server_negotiation.cc:407] Connection header received
0925 15:20:33.588447 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:33.588448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:33.588464 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:33.588539 (+ 75us) server_negotiation.cc:226] Negotiated authn=JWT
0925 15:20:33.588692 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:33.589511 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:33.590379 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:33.590692 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:33.590766 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:33.590960 (+ 194us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 15:20:33.604421 (+ 13461us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36979/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 15:20:33.604598 (+ 177us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59672: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36979/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":41}
E20260925 15:20:33.604698 9131 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260925 15:20:33.604759 9104 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59672: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36979/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260925 15:20:33.605361 9130 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:33.588025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:41095 (local address 127.0.0.1:59672)
0925 15:20:33.588227 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:33.588237 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:33.588299 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:33.588556 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:33.588557 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:33.588559 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 15:20:33.588623 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:33.588625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:33.589635 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:33.589636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:33.590278 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:33.590280 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:33.590773 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:33.590774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:33.590823 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:33.590844 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 15:20:33.604590 (+ 13746us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36979/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 15:20:33.605106 (+ 516us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.211.254:41095: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36979/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":87,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.3.211.254:41095: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36979/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 15:20:33.605854 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 9034
2026-09-25T15:20:33Z chronyd exiting
I20260925 15:20:33.642069 3919 test_util.cc:175] -----------------------------------------------
I20260925 15:20:33.642198 3919 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790349610595785-3919-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (246 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 15:20:33.643029 3919 mini_oidc.cc:152] Starting JWKS server
I20260925 15:20:33.643765 3919 webserver.cc:460] Webserver started at https://127.0.0.1:36407/ using document root <none> and password file <none>
I20260925 15:20:33.643842 3919 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 15:20:33.644140 3919 webserver.cc:460] Webserver started at http://127.0.0.1:38283/ using document root <none> and password file <none>
2026-09-25T15:20:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:20:33Z Disabled control of system clock
I20260925 15:20:33.652266 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:34543
--webserver_interface=127.3.211.254
--webserver_port=0
--builtin_ntp_servers=127.3.211.212:36159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.211.254:34543
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36407/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349610595785-3919-0/testchainca.cert with env {}
W20260925 15:20:33.739310 9137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:33.739573 9137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:33.739600 9137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:33.740953 9137 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:20:33.741020 9137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:33.741034 9137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:33.741047 9137 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:20:33.741065 9137 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 15:20:33.741078 9137 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 15:20:33.741091 9137 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36407/jwks.json
I20260925 15:20:33.742808 9137 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:36159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349610595785-3919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.211.254:34543
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:34543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36407/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.211.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349610595785-3919-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:33.743057 9137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:33.743238 9137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:33.748389 9143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:33.748371 9142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:33.748426 9145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:33.748553 9137 server_base.cc:1034] running on GCE node
I20260925 15:20:33.748881 9137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:33.749284 9137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:33.750468 9137 hybrid_clock.cc:648] HybridClock initialized: now 1790349633750441 us; error 39 us; skew 500 ppm
I20260925 15:20:33.752743 9137 webserver.cc:460] Webserver started at http://127.3.211.254:41843/ using document root <none> and password file <none>
I20260925 15:20:33.753062 9137 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:33.753123 9137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:33.753275 9137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:33.755097 9137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349610595785-3919-0/minicluster-data/master-0/data/instance:
uuid: "f551ec8008dd4760b74795fd0c0da900"
format_stamp: "Formatted at 2026-09-25 15:20:33 on dist-test-slave-4bk3"
server_key: "3d241cbf5eedac71d8faaa983fc8dd5c"
server_key_iv: "81d89b0730c6907de19c93d73f4fa7d6"
server_key_version: "encryptionkey@0"
I20260925 15:20:33.755551 9137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349610595785-3919-0/minicluster-data/master-0/wal/instance:
uuid: "f551ec8008dd4760b74795fd0c0da900"
format_stamp: "Formatted at 2026-09-25 15:20:33 on dist-test-slave-4bk3"
server_key: "3d241cbf5eedac71d8faaa983fc8dd5c"
server_key_iv: "81d89b0730c6907de19c93d73f4fa7d6"
server_key_version: "encryptionkey@0"
I20260925 15:20:33.757230 9137 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260925 15:20:33.758271 9151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:33.758641 9137 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:20:33.758754 9137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349610595785-3919-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349610595785-3919-0/minicluster-data/master-0/wal
uuid: "f551ec8008dd4760b74795fd0c0da900"
format_stamp: "Formatted at 2026-09-25 15:20:33 on dist-test-slave-4bk3"
server_key: "3d241cbf5eedac71d8faaa983fc8dd5c"
server_key_iv: "81d89b0730c6907de19c93d73f4fa7d6"
server_key_version: "encryptionkey@0"
I20260925 15:20:33.758870 9137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349610595785-3919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349610595785-3919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349610595785-3919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:33.787397 9137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:33.787789 9137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:33.787958 9137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:33.792116 9137 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.254:34543
I20260925 15:20:33.792196 9203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.254:34543 every 8 connection(s)
I20260925 15:20:33.792575 9137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
I20260925 15:20:33.793586 9204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:33.796082 9204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f551ec8008dd4760b74795fd0c0da900: Bootstrap starting.
I20260925 15:20:33.796809 9204 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f551ec8008dd4760b74795fd0c0da900: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:33.797070 9204 log.cc:824] T 00000000000000000000000000000000 P f551ec8008dd4760b74795fd0c0da900: Log is configured to *not* fsync() on all Append() calls
I20260925 15:20:33.797821 9204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f551ec8008dd4760b74795fd0c0da900: No bootstrap required, opened a new log
I20260925 15:20:33.798024 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 9137
I20260925 15:20:33.798130 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790349610595785-3919-0/minicluster-data/master-0/wal/instance
I20260925 15:20:33.798306 3919 external_mini_cluster.cc:1428] Setting key 170e369574c7865bf2d080b215e2f776
I20260925 15:20:33.799907 9204 raft_consensus.cc:348] T 00000000000000000000000000000000 P f551ec8008dd4760b74795fd0c0da900 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f551ec8008dd4760b74795fd0c0da900" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 34543 } }
I20260925 15:20:33.800102 9204 raft_consensus.cc:374] T 00000000000000000000000000000000 P f551ec8008dd4760b74795fd0c0da900 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:33.800141 9204 raft_consensus.cc:729] T 00000000000000000000000000000000 P f551ec8008dd4760b74795fd0c0da900 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f551ec8008dd4760b74795fd0c0da900, State: Initialized, Role: FOLLOWER
I20260925 15:20:33.800307 9204 consensus_queue.cc:260] T 00000000000000000000000000000000 P f551ec8008dd4760b74795fd0c0da900 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f551ec8008dd4760b74795fd0c0da900" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 34543 } }
I20260925 15:20:33.800436 9204 raft_consensus.cc:388] T 00000000000000000000000000000000 P f551ec8008dd4760b74795fd0c0da900 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:20:33.800484 9204 raft_consensus.cc:482] T 00000000000000000000000000000000 P f551ec8008dd4760b74795fd0c0da900 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:20:33.800536 9204 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f551ec8008dd4760b74795fd0c0da900 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:33.801136 9204 raft_consensus.cc:504] T 00000000000000000000000000000000 P f551ec8008dd4760b74795fd0c0da900 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f551ec8008dd4760b74795fd0c0da900" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 34543 } }
I20260925 15:20:33.801259 9204 leader_election.cc:302] T 00000000000000000000000000000000 P f551ec8008dd4760b74795fd0c0da900 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f551ec8008dd4760b74795fd0c0da900; no voters:
I20260925 15:20:33.801465 9204 leader_election.cc:288] T 00000000000000000000000000000000 P f551ec8008dd4760b74795fd0c0da900 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:20:33.801494 9209 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f551ec8008dd4760b74795fd0c0da900 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:20:33.801602 9209 raft_consensus.cc:686] T 00000000000000000000000000000000 P f551ec8008dd4760b74795fd0c0da900 [term 1 LEADER]: Becoming Leader. State: Replica: f551ec8008dd4760b74795fd0c0da900, State: Running, Role: LEADER
I20260925 15:20:33.801685 9209 consensus_queue.cc:237] T 00000000000000000000000000000000 P f551ec8008dd4760b74795fd0c0da900 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f551ec8008dd4760b74795fd0c0da900" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 34543 } }
I20260925 15:20:33.801755 9204 sys_catalog.cc:564] T 00000000000000000000000000000000 P f551ec8008dd4760b74795fd0c0da900 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:20:33.802030 9210 sys_catalog.cc:455] T 00000000000000000000000000000000 P f551ec8008dd4760b74795fd0c0da900 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f551ec8008dd4760b74795fd0c0da900" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f551ec8008dd4760b74795fd0c0da900" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 34543 } } }
I20260925 15:20:33.802150 9209 sys_catalog.cc:455] T 00000000000000000000000000000000 P f551ec8008dd4760b74795fd0c0da900 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f551ec8008dd4760b74795fd0c0da900. Latest consensus state: current_term: 1 leader_uuid: "f551ec8008dd4760b74795fd0c0da900" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f551ec8008dd4760b74795fd0c0da900" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 34543 } } }
I20260925 15:20:33.802258 9209 sys_catalog.cc:458] T 00000000000000000000000000000000 P f551ec8008dd4760b74795fd0c0da900 [sys.catalog]: This master's current role is: LEADER
I20260925 15:20:33.802186 9210 sys_catalog.cc:458] T 00000000000000000000000000000000 P f551ec8008dd4760b74795fd0c0da900 [sys.catalog]: This master's current role is: LEADER
I20260925 15:20:33.802496 9214 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:20:33.803025 9214 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:20:33.805095 9214 catalog_manager.cc:1347] Generated new cluster ID: bd9b92e74b654baba99b3248c71bedb8
I20260925 15:20:33.805233 9214 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:20:33.805495 9208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:33.798825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44896 (local address 127.3.211.254:34543)
0925 15:20:33.799046 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:33.799053 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:33.799079 (+ 26us) server_negotiation.cc:407] Connection header received
0925 15:20:33.799906 (+ 827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:33.799914 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:33.799985 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:33.800163 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:33.800414 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:33.802006 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:33.803188 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:33.803774 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:33.803926 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:33.804716 (+ 790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:33.804719 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:33.804719 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:20:33.804927 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:33.804970 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:20:33.804983 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:33.805051 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:33.805202 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 15:20:33.805290 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":97,"threads_started":1}
I20260925 15:20:33.806501 3919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:20:33.818189 9214 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:20:33.818918 9214 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:20:33.823940 9214 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f551ec8008dd4760b74795fd0c0da900: Generated new TSK 0
I20260925 15:20:33.824162 9214 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 15:20:33.847898 9234 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 15:20:33.848065 9234 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 15:20:33.848227 9208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:33.832023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44906 (local address 127.3.211.254:34543)
0925 15:20:33.832124 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:33.832126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:33.832141 (+ 15us) server_negotiation.cc:407] Connection header received
0925 15:20:33.832217 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:33.832218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:33.832231 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:33.832364 (+ 133us) server_negotiation.cc:226] Negotiated authn=JWT
0925 15:20:33.832554 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:33.833328 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:33.834111 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:33.834435 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:33.834513 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:33.834739 (+ 226us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 15:20:33.847976 (+ 13237us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36407/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 15:20:33.848147 (+ 171us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44906: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36407/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":42}
W20260925 15:20:33.848299 9208 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44906: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36407/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260925 15:20:33.848526 9233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:33.831790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34543 (local address 127.0.0.1:44906)
0925 15:20:33.832093 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:33.832103 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:33.832150 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:33.832404 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:33.832405 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:33.832407 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 15:20:33.832473 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:33.832475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:33.833421 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:33.833421 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:33.834020 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:33.834021 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:33.834549 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:33.834550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:33.834603 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:33.834610 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 15:20:33.848244 (+ 13634us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36407/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 15:20:33.848432 (+ 188us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.211.254:34543: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36407/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":220,"threads_started":1}
I20260925 15:20:33.848881 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 9137
2026-09-25T15:20:33Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (243 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 15:20:33.886739 3919 mini_oidc.cc:152] Starting JWKS server
I20260925 15:20:33.887411 3919 webserver.cc:460] Webserver started at https://127.0.0.1:38681/ using document root <none> and password file <none>
I20260925 15:20:33.887456 3919 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 15:20:33.887619 3919 webserver.cc:460] Webserver started at http://127.0.0.1:37915/ using document root <none> and password file <none>
2026-09-25T15:20:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:20:33Z Disabled control of system clock
I20260925 15:20:33.895735 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:37359
--webserver_interface=127.3.211.254
--webserver_port=0
--builtin_ntp_servers=127.3.211.212:37357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.211.254:37359
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38681/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 15:20:33.978507 9240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:33.978812 9240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:33.978861 9240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:33.980202 9240 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:20:33.980289 9240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:33.980311 9240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:33.980332 9240 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:20:33.980353 9240 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 15:20:33.980429 9240 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 15:20:33.980458 9240 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38681/jwks.json
I20260925 15:20:33.981985 9240 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:37357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349610595785-3919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.211.254:37359
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:37359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38681/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.211.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:33.982308 9240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:33.982633 9240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:33.987773 9245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:33.987779 9248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:33.987850 9246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:33.988030 9240 server_base.cc:1034] running on GCE node
I20260925 15:20:33.988296 9240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:33.988692 9240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:33.989876 9240 hybrid_clock.cc:648] HybridClock initialized: now 1790349633989858 us; error 32 us; skew 500 ppm
I20260925 15:20:33.991230 9240 webserver.cc:460] Webserver started at http://127.3.211.254:45257/ using document root <none> and password file <none>
I20260925 15:20:33.991462 9240 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:33.991508 9240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:33.991605 9240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:33.992810 9240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349610595785-3919-0/minicluster-data/master-0/data/instance:
uuid: "c2f464fc329549a5a722de9e8fab0af0"
format_stamp: "Formatted at 2026-09-25 15:20:33 on dist-test-slave-4bk3"
server_key: "3857f122ee522a4d24180a81d2f871ee"
server_key_iv: "ceff9d5bd7c9a956767b113e0d639eb4"
server_key_version: "encryptionkey@0"
I20260925 15:20:33.993109 9240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349610595785-3919-0/minicluster-data/master-0/wal/instance:
uuid: "c2f464fc329549a5a722de9e8fab0af0"
format_stamp: "Formatted at 2026-09-25 15:20:33 on dist-test-slave-4bk3"
server_key: "3857f122ee522a4d24180a81d2f871ee"
server_key_iv: "ceff9d5bd7c9a956767b113e0d639eb4"
server_key_version: "encryptionkey@0"
I20260925 15:20:33.994493 9240 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:20:33.995298 9254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:33.995507 9240 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:33.995675 9240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349610595785-3919-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349610595785-3919-0/minicluster-data/master-0/wal
uuid: "c2f464fc329549a5a722de9e8fab0af0"
format_stamp: "Formatted at 2026-09-25 15:20:33 on dist-test-slave-4bk3"
server_key: "3857f122ee522a4d24180a81d2f871ee"
server_key_iv: "ceff9d5bd7c9a956767b113e0d639eb4"
server_key_version: "encryptionkey@0"
I20260925 15:20:33.995766 9240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349610595785-3919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349610595785-3919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349610595785-3919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:34.030395 9240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:34.030810 9240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:34.030980 9240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:34.035154 9240 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.254:37359
I20260925 15:20:34.035195 9306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.254:37359 every 8 connection(s)
I20260925 15:20:34.035554 9240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
I20260925 15:20:34.036245 9307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:34.038980 9307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2f464fc329549a5a722de9e8fab0af0: Bootstrap starting.
I20260925 15:20:34.039515 9307 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2f464fc329549a5a722de9e8fab0af0: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:34.039745 9307 log.cc:824] T 00000000000000000000000000000000 P c2f464fc329549a5a722de9e8fab0af0: Log is configured to *not* fsync() on all Append() calls
I20260925 15:20:34.040544 9307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2f464fc329549a5a722de9e8fab0af0: No bootstrap required, opened a new log
I20260925 15:20:34.040990 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 9240
I20260925 15:20:34.041095 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790349610595785-3919-0/minicluster-data/master-0/wal/instance
I20260925 15:20:34.041262 3919 external_mini_cluster.cc:1428] Setting key 127ddb08c47800670e3220abf8d25bc4
I20260925 15:20:34.042738 9307 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2f464fc329549a5a722de9e8fab0af0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2f464fc329549a5a722de9e8fab0af0" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 37359 } }
I20260925 15:20:34.042872 9307 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2f464fc329549a5a722de9e8fab0af0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:34.042896 9307 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2f464fc329549a5a722de9e8fab0af0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2f464fc329549a5a722de9e8fab0af0, State: Initialized, Role: FOLLOWER
I20260925 15:20:34.043025 9307 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2f464fc329549a5a722de9e8fab0af0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2f464fc329549a5a722de9e8fab0af0" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 37359 } }
I20260925 15:20:34.043104 9307 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2f464fc329549a5a722de9e8fab0af0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:20:34.043139 9307 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2f464fc329549a5a722de9e8fab0af0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:20:34.043192 9307 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2f464fc329549a5a722de9e8fab0af0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:34.043756 9307 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2f464fc329549a5a722de9e8fab0af0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2f464fc329549a5a722de9e8fab0af0" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 37359 } }
I20260925 15:20:34.043841 9307 leader_election.cc:302] T 00000000000000000000000000000000 P c2f464fc329549a5a722de9e8fab0af0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2f464fc329549a5a722de9e8fab0af0; no voters:
I20260925 15:20:34.044025 9307 leader_election.cc:288] T 00000000000000000000000000000000 P c2f464fc329549a5a722de9e8fab0af0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:20:34.044055 9312 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2f464fc329549a5a722de9e8fab0af0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:20:34.044178 9312 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2f464fc329549a5a722de9e8fab0af0 [term 1 LEADER]: Becoming Leader. State: Replica: c2f464fc329549a5a722de9e8fab0af0, State: Running, Role: LEADER
I20260925 15:20:34.044302 9307 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2f464fc329549a5a722de9e8fab0af0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:20:34.044281 9312 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2f464fc329549a5a722de9e8fab0af0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2f464fc329549a5a722de9e8fab0af0" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 37359 } }
I20260925 15:20:34.044793 9313 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2f464fc329549a5a722de9e8fab0af0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2f464fc329549a5a722de9e8fab0af0. Latest consensus state: current_term: 1 leader_uuid: "c2f464fc329549a5a722de9e8fab0af0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2f464fc329549a5a722de9e8fab0af0" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 37359 } } }
I20260925 15:20:34.045029 9313 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2f464fc329549a5a722de9e8fab0af0 [sys.catalog]: This master's current role is: LEADER
I20260925 15:20:34.044804 9314 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2f464fc329549a5a722de9e8fab0af0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2f464fc329549a5a722de9e8fab0af0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2f464fc329549a5a722de9e8fab0af0" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 37359 } } }
I20260925 15:20:34.045192 9314 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2f464fc329549a5a722de9e8fab0af0 [sys.catalog]: This master's current role is: LEADER
W20260925 15:20:34.046151 9327 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c2f464fc329549a5a722de9e8fab0af0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:20:34.046221 9327 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:20:34.046284 9321 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:20:34.046468 9321 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:20:34.048645 9321 catalog_manager.cc:1347] Generated new cluster ID: b3d9cfce8d164cccbaba8fd491d69dab
I20260925 15:20:34.048708 9311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:34.041781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45584 (local address 127.3.211.254:37359)
0925 15:20:34.041984 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:34.041992 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:34.042016 (+ 24us) server_negotiation.cc:407] Connection header received
0925 15:20:34.042750 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:34.042758 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:34.042820 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:34.043005 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:34.043278 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:34.045024 (+ 1746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:34.046206 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:34.046807 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:34.046929 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:34.047770 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:34.047774 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:34.047774 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:20:34.047987 (+ 213us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:34.048033 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:20:34.048048 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:34.048157 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:34.048246 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 15:20:34.048459 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":98,"threads_started":1}
I20260925 15:20:34.048720 9321 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:20:34.049777 3919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:20:34.061625 9321 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:20:34.062347 9321 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:20:34.066824 9321 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2f464fc329549a5a722de9e8fab0af0: Generated new TSK 0
I20260925 15:20:34.067085 9321 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:20:34.079047 9311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:34.074849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45590 (local address 127.3.211.254:37359)
0925 15:20:34.074916 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:34.074918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:34.075029 (+ 111us) server_negotiation.cc:407] Connection header received
0925 15:20:34.075117 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:34.075118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:34.075144 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:34.075230 (+ 86us) server_negotiation.cc:226] Negotiated authn=JWT
0925 15:20:34.075427 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:34.076216 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:34.077147 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:34.077483 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:34.077563 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:34.077736 (+ 173us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 15:20:34.078801 (+ 1065us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38681/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 15:20:34.078955 (+ 154us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45590: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38681/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":39}
W20260925 15:20:34.079119 9311 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45590: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38681/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260925 15:20:34.081091 9336 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:34.074750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:37359 (local address 127.0.0.1:45590)
0925 15:20:34.074947 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:34.074960 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:34.075025 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:34.075255 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:34.075257 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:34.075259 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 15:20:34.075343 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:34.075345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:34.076403 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:34.076405 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:34.077049 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:34.077052 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:34.077572 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:34.077573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:34.077620 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:34.077626 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 15:20:34.078921 (+ 1295us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38681/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 15:20:34.080986 (+ 2065us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.211.254:37359: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38681/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":1811,"thread_start_us":120,"threads_started":1}
I20260925 15:20:34.081471 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 9240
E20260925 15:20:34.091842 9337 webserver.cc:533] Webserver: sslize failed:
E20260925 15:20:34.091930 9337 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-25T15:20:34Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (233 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T15:20:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:20:34Z Disabled control of system clock
I20260925 15:20:34.128839 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:37875
--webserver_interface=127.3.211.254
--webserver_port=0
--builtin_ntp_servers=127.3.211.212:46533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.211.254:37875
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349610595785-3919-0/testchainca.cert with env {}
W20260925 15:20:34.214803 9341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:34.215046 9341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:34.215071 9341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:34.216434 9341 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:20:34.216487 9341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:34.216500 9341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:34.216513 9341 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:20:34.216526 9341 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 15:20:34.216538 9341 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 15:20:34.216578 9341 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 15:20:34.218299 9341 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:46533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349610595785-3919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.211.254:37875
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:37875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.211.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349610595785-3919-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:34.218609 9341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:34.218840 9341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:34.224061 9349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:34.224157 9347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:34.224365 9341 server_base.cc:1034] running on GCE node
W20260925 15:20:34.224159 9346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:34.224776 9341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:34.225102 9341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:34.226305 9341 hybrid_clock.cc:648] HybridClock initialized: now 1790349634226293 us; error 37 us; skew 500 ppm
I20260925 15:20:34.228178 9341 webserver.cc:460] Webserver started at http://127.3.211.254:37041/ using document root <none> and password file <none>
I20260925 15:20:34.228499 9341 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:34.228556 9341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:34.228701 9341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:34.229851 9341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349610595785-3919-0/minicluster-data/master-0/data/instance:
uuid: "ed5049e01c5f429d978b2adb40f93628"
format_stamp: "Formatted at 2026-09-25 15:20:34 on dist-test-slave-4bk3"
server_key: "8500961d743c4616d92162e20297aecc"
server_key_iv: "08d090ae9d1ea4146f97d267cfe89d55"
server_key_version: "encryptionkey@0"
I20260925 15:20:34.230242 9341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349610595785-3919-0/minicluster-data/master-0/wal/instance:
uuid: "ed5049e01c5f429d978b2adb40f93628"
format_stamp: "Formatted at 2026-09-25 15:20:34 on dist-test-slave-4bk3"
server_key: "8500961d743c4616d92162e20297aecc"
server_key_iv: "08d090ae9d1ea4146f97d267cfe89d55"
server_key_version: "encryptionkey@0"
I20260925 15:20:34.232131 9341 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:20:34.233292 9355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:34.233520 9341 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:34.233629 9341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349610595785-3919-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349610595785-3919-0/minicluster-data/master-0/wal
uuid: "ed5049e01c5f429d978b2adb40f93628"
format_stamp: "Formatted at 2026-09-25 15:20:34 on dist-test-slave-4bk3"
server_key: "8500961d743c4616d92162e20297aecc"
server_key_iv: "08d090ae9d1ea4146f97d267cfe89d55"
server_key_version: "encryptionkey@0"
I20260925 15:20:34.233716 9341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349610595785-3919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349610595785-3919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349610595785-3919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:34.265885 9341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:34.266315 9341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:34.266485 9341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:34.271126 9341 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.254:37875
I20260925 15:20:34.271176 9407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.254:37875 every 8 connection(s)
I20260925 15:20:34.271593 9341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
I20260925 15:20:34.272642 9408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:34.273959 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 9341
I20260925 15:20:34.274086 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790349610595785-3919-0/minicluster-data/master-0/wal/instance
I20260925 15:20:34.274273 3919 external_mini_cluster.cc:1428] Setting key af2abc375e166c3cf30b48c828bd84e6
I20260925 15:20:34.275432 9408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed5049e01c5f429d978b2adb40f93628: Bootstrap starting.
I20260925 15:20:34.276196 9408 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed5049e01c5f429d978b2adb40f93628: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:34.276553 9408 log.cc:824] T 00000000000000000000000000000000 P ed5049e01c5f429d978b2adb40f93628: Log is configured to *not* fsync() on all Append() calls
I20260925 15:20:34.277480 9408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed5049e01c5f429d978b2adb40f93628: No bootstrap required, opened a new log
I20260925 15:20:34.279990 9408 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed5049e01c5f429d978b2adb40f93628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5049e01c5f429d978b2adb40f93628" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 37875 } }
I20260925 15:20:34.280153 9408 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed5049e01c5f429d978b2adb40f93628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:34.280175 9408 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed5049e01c5f429d978b2adb40f93628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed5049e01c5f429d978b2adb40f93628, State: Initialized, Role: FOLLOWER
I20260925 15:20:34.280310 9408 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed5049e01c5f429d978b2adb40f93628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5049e01c5f429d978b2adb40f93628" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 37875 } }
I20260925 15:20:34.280429 9408 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed5049e01c5f429d978b2adb40f93628 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:20:34.280474 9408 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed5049e01c5f429d978b2adb40f93628 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:20:34.280526 9408 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed5049e01c5f429d978b2adb40f93628 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:34.280833 9411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:34.274857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43674 (local address 127.3.211.254:37875)
0925 15:20:34.275051 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:34.275064 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:34.275090 (+ 26us) server_negotiation.cc:407] Connection header received
0925 15:20:34.275478 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:34.275485 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:34.275575 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:34.275717 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:34.276009 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:34.277237 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:34.278474 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:34.278974 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:34.279094 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:34.279922 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:34.279926 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:34.279926 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:20:34.280107 (+ 181us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:34.280187 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:20:34.280204 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:34.280278 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:34.280452 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 15:20:34.280651 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":81,"threads_started":1}
I20260925 15:20:34.281235 9408 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed5049e01c5f429d978b2adb40f93628 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5049e01c5f429d978b2adb40f93628" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 37875 } }
I20260925 15:20:34.281334 9408 leader_election.cc:302] T 00000000000000000000000000000000 P ed5049e01c5f429d978b2adb40f93628 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed5049e01c5f429d978b2adb40f93628; no voters:
I20260925 15:20:34.281520 9408 leader_election.cc:288] T 00000000000000000000000000000000 P ed5049e01c5f429d978b2adb40f93628 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:20:34.281656 9413 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed5049e01c5f429d978b2adb40f93628 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:20:34.281821 9408 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed5049e01c5f429d978b2adb40f93628 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:20:34.281842 9413 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed5049e01c5f429d978b2adb40f93628 [term 1 LEADER]: Becoming Leader. State: Replica: ed5049e01c5f429d978b2adb40f93628, State: Running, Role: LEADER
I20260925 15:20:34.281936 9413 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed5049e01c5f429d978b2adb40f93628 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5049e01c5f429d978b2adb40f93628" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 37875 } }
I20260925 15:20:34.281967 3919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:20:34.282286 9413 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed5049e01c5f429d978b2adb40f93628 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed5049e01c5f429d978b2adb40f93628. Latest consensus state: current_term: 1 leader_uuid: "ed5049e01c5f429d978b2adb40f93628" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5049e01c5f429d978b2adb40f93628" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 37875 } } }
I20260925 15:20:34.282354 9413 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed5049e01c5f429d978b2adb40f93628 [sys.catalog]: This master's current role is: LEADER
I20260925 15:20:34.282292 9414 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed5049e01c5f429d978b2adb40f93628 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed5049e01c5f429d978b2adb40f93628" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed5049e01c5f429d978b2adb40f93628" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 37875 } } }
I20260925 15:20:34.282402 9414 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed5049e01c5f429d978b2adb40f93628 [sys.catalog]: This master's current role is: LEADER
I20260925 15:20:34.283176 9420 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:20:34.283872 9420 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:20:34.285547 9420 catalog_manager.cc:1347] Generated new cluster ID: 8db5792ca20f4791a2de4443adf29360
I20260925 15:20:34.285610 9420 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:20:34.294150 9420 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:20:34.294721 9420 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:20:34.303835 9420 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed5049e01c5f429d978b2adb40f93628: Generated new TSK 0
I20260925 15:20:34.304169 9420 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:20:34.418488 9411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:34.315491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43690 (local address 127.3.211.254:37875)
0925 15:20:34.315543 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:34.315551 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:34.315698 (+ 147us) server_negotiation.cc:407] Connection header received
0925 15:20:34.315767 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:34.315768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:34.315779 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:34.315853 (+ 74us) server_negotiation.cc:226] Negotiated authn=JWT
0925 15:20:34.316020 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:34.316603 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:34.317848 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:34.318056 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:34.318121 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:34.318500 (+ 379us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 15:20:34.418244 (+ 99744us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 15:20:34.418425 (+ 181us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43690: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":29}
W20260925 15:20:34.418583 9411 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43690: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260925 15:20:34.418747 9437 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:34.315337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:37875 (local address 127.0.0.1:43690)
0925 15:20:34.315555 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:34.315568 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:34.315685 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:34.315869 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:34.315870 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:34.315873 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0925 15:20:34.315940 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:34.315943 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:34.316720 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:34.316721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:34.317733 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:34.317736 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:34.318150 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:34.318151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:34.318211 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:34.318217 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 15:20:34.418454 (+100237us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 15:20:34.418636 (+ 182us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.211.254:37875: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":144,"threads_started":1}
I20260925 15:20:34.419071 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 9341
2026-09-25T15:20:34Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (316 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (93 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (95 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:20:34 dist-test-slave-4bk3 krb5kdc[9449](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:20:34 dist-test-slave-4bk3 krb5kdc[9449](info): set up 2 sockets
Sep 25 15:20:34 dist-test-slave-4bk3 krb5kdc[9449](info): commencing operation
krb5kdc: starting...
W20260925 15:20:35.660588 3919 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Sep 25 15:20:35 dist-test-slave-4bk3 krb5kdc[9449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:20:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:20:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.211.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
I20260925 15:20:35.788313 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:33169
--webserver_interface=127.3.211.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:46761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.211.254:33169
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:20:35.878909 9462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:35.879251 9462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:35.879297 9462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:35.880858 9462 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:20:35.880929 9462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:35.880954 9462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:35.880986 9462 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:20:35.881013 9462 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:20:35.883258 9462 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:46761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.211.254:33169
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:33169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.211.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:35.883577 9462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:35.883880 9462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:35.889540 9467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:35.889701 9462 server_base.cc:1034] running on GCE node
W20260925 15:20:35.889703 9470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:35.889535 9468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:35.890070 9462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:35.890357 9462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:35.891518 9462 hybrid_clock.cc:648] HybridClock initialized: now 1790349635891494 us; error 33 us; skew 500 ppm
Sep 25 15:20:35 dist-test-slave-4bk3 krb5kdc[9449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349635, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:35.895627 9462 init.cc:377] Logged in from keytab as kudu/127.3.211.254@KRBTEST.COM (short username kudu)
I20260925 15:20:35.896369 9462 webserver.cc:460] Webserver started at http://127.3.211.254:38163/ using document root <none> and password file <none>
I20260925 15:20:35.896649 9462 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:35.896718 9462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:35.896871 9462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:35.898485 9462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/master-0/data/instance:
uuid: "ab4cb8abdbad429c82765f53cdfaf79e"
format_stamp: "Formatted at 2026-09-25 15:20:35 on dist-test-slave-4bk3"
server_key: "178ad09e9f56e39625d7b9b994137a73"
server_key_iv: "98bd8c391a0a05af77e9177aaa8c7f3e"
server_key_version: "encryptionkey@0"
I20260925 15:20:35.898938 9462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/master-0/wal/instance:
uuid: "ab4cb8abdbad429c82765f53cdfaf79e"
format_stamp: "Formatted at 2026-09-25 15:20:35 on dist-test-slave-4bk3"
server_key: "178ad09e9f56e39625d7b9b994137a73"
server_key_iv: "98bd8c391a0a05af77e9177aaa8c7f3e"
server_key_version: "encryptionkey@0"
I20260925 15:20:35.901051 9462 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260925 15:20:35.902287 9477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:35.902468 9462 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:35.902596 9462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/master-0/wal
uuid: "ab4cb8abdbad429c82765f53cdfaf79e"
format_stamp: "Formatted at 2026-09-25 15:20:35 on dist-test-slave-4bk3"
server_key: "178ad09e9f56e39625d7b9b994137a73"
server_key_iv: "98bd8c391a0a05af77e9177aaa8c7f3e"
server_key_version: "encryptionkey@0"
I20260925 15:20:35.902691 9462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:35.916002 9462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:35.918533 9462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:35.918715 9462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:35.922851 9462 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.254:33169
I20260925 15:20:35.922884 9529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.254:33169 every 8 connection(s)
I20260925 15:20:35.923322 9462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
I20260925 15:20:35.923501 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 9462
I20260925 15:20:35.923610 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/master-0/wal/instance
I20260925 15:20:35.923808 3919 external_mini_cluster.cc:1428] Setting key 3da0fab4b57cc9bc0ffd9393be395059
I20260925 15:20:35.924546 9530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:35.928233 9530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab4cb8abdbad429c82765f53cdfaf79e: Bootstrap starting.
I20260925 15:20:35.928974 9530 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab4cb8abdbad429c82765f53cdfaf79e: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:35.929320 9530 log.cc:824] T 00000000000000000000000000000000 P ab4cb8abdbad429c82765f53cdfaf79e: Log is configured to *not* fsync() on all Append() calls
I20260925 15:20:35.930173 9530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab4cb8abdbad429c82765f53cdfaf79e: No bootstrap required, opened a new log
Sep 25 15:20:35 dist-test-slave-4bk3 krb5kdc[9449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:20:35.932689 9530 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab4cb8abdbad429c82765f53cdfaf79e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab4cb8abdbad429c82765f53cdfaf79e" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 33169 } }
I20260925 15:20:35.932914 9530 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab4cb8abdbad429c82765f53cdfaf79e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:35.932965 9530 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab4cb8abdbad429c82765f53cdfaf79e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab4cb8abdbad429c82765f53cdfaf79e, State: Initialized, Role: FOLLOWER
I20260925 15:20:35.933117 9530 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab4cb8abdbad429c82765f53cdfaf79e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab4cb8abdbad429c82765f53cdfaf79e" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 33169 } }
I20260925 15:20:35.933216 9530 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab4cb8abdbad429c82765f53cdfaf79e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:20:35.933252 9530 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab4cb8abdbad429c82765f53cdfaf79e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:20:35.933305 9530 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab4cb8abdbad429c82765f53cdfaf79e [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:35.934126 9530 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab4cb8abdbad429c82765f53cdfaf79e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab4cb8abdbad429c82765f53cdfaf79e" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 33169 } }
I20260925 15:20:35.934234 9530 leader_election.cc:302] T 00000000000000000000000000000000 P ab4cb8abdbad429c82765f53cdfaf79e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab4cb8abdbad429c82765f53cdfaf79e; no voters:
I20260925 15:20:35.934433 9530 leader_election.cc:288] T 00000000000000000000000000000000 P ab4cb8abdbad429c82765f53cdfaf79e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:20:35.934470 9535 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab4cb8abdbad429c82765f53cdfaf79e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:20:35.934626 9535 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab4cb8abdbad429c82765f53cdfaf79e [term 1 LEADER]: Becoming Leader. State: Replica: ab4cb8abdbad429c82765f53cdfaf79e, State: Running, Role: LEADER
I20260925 15:20:35.934717 9530 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab4cb8abdbad429c82765f53cdfaf79e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:20:35.934754 9535 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab4cb8abdbad429c82765f53cdfaf79e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab4cb8abdbad429c82765f53cdfaf79e" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 33169 } }
I20260925 15:20:35.935101 9535 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab4cb8abdbad429c82765f53cdfaf79e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab4cb8abdbad429c82765f53cdfaf79e. Latest consensus state: current_term: 1 leader_uuid: "ab4cb8abdbad429c82765f53cdfaf79e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab4cb8abdbad429c82765f53cdfaf79e" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 33169 } } }
I20260925 15:20:35.935175 9535 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab4cb8abdbad429c82765f53cdfaf79e [sys.catalog]: This master's current role is: LEADER
I20260925 15:20:35.936201 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:35.924393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42000 (local address 127.3.211.254:33169)
0925 15:20:35.924577 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:35.924586 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:35.924638 (+ 52us) server_negotiation.cc:407] Connection header received
0925 15:20:35.925100 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:35.925106 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:35.925152 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:35.925269 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:35.926199 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:35.927311 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:35.928504 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:35.928969 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:35.929089 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:35.931856 (+ 2767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:35.931861 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:35.931861 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:35.931876 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:35.933990 (+ 2114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:35.934413 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:35.934417 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:35.934417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:35.934529 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:35.934775 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:35.934776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:35.934776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:35.935113 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:35.935281 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:35.935770 (+ 489us) server_negotiation.cc:299] Negotiation successful
0925 15:20:35.935938 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":96,"threads_started":1}
W20260925 15:20:35.936643 9551 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ab4cb8abdbad429c82765f53cdfaf79e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:20:35.936705 9551 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:20:35.936677 9536 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab4cb8abdbad429c82765f53cdfaf79e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab4cb8abdbad429c82765f53cdfaf79e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab4cb8abdbad429c82765f53cdfaf79e" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 33169 } } }
I20260925 15:20:35.936748 9536 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab4cb8abdbad429c82765f53cdfaf79e [sys.catalog]: This master's current role is: LEADER
I20260925 15:20:35.936759 9541 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:20:35.936910 9541 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:20:35.938346 9541 catalog_manager.cc:1347] Generated new cluster ID: 70c11f36500f481e9ff9c0b27dd81c6d
I20260925 15:20:35.938405 9541 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:20:35.970886 9541 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:20:35.971499 9541 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:20:35.976883 9541 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab4cb8abdbad429c82765f53cdfaf79e: Generated new TSK 0
I20260925 15:20:35.977078 9541 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.211.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab.
Entry for principal kudu/127.3.211.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab.
I20260925 15:20:36.012238 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.193:0
--local_ip_for_outbound_sockets=127.3.211.193
--webserver_interface=127.3.211.193
--webserver_port=0
--tserver_master_addrs=127.3.211.254:33169
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab
--principal=kudu/127.3.211.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:46761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:20:36.095886 9556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:36.096107 9556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:36.096130 9556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:36.097383 9556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:36.097432 9556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:36.097461 9556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.193
I20260925 15:20:36.099046 9556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:46761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab
--principal=kudu/127.3.211.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.211.193
--webserver_port=0
--tserver_master_addrs=127.3.211.254:33169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.193
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:36.099325 9556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:36.099573 9556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:36.104760 9564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:36.104760 9562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:36.104760 9561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:36.105085 9556 server_base.cc:1034] running on GCE node
I20260925 15:20:36.105351 9556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:36.105638 9556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:36.106838 9556 hybrid_clock.cc:648] HybridClock initialized: now 1790349636106799 us; error 52 us; skew 500 ppm
Sep 25 15:20:36 dist-test-slave-4bk3 krb5kdc[9449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349636, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:36.109678 9556 init.cc:377] Logged in from keytab as kudu/127.3.211.193@KRBTEST.COM (short username kudu)
I20260925 15:20:36.110155 9556 webserver.cc:460] Webserver started at http://127.3.211.193:41957/ using document root <none> and password file <none>
I20260925 15:20:36.110409 9556 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:36.110462 9556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:36.110596 9556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:36.111709 9556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/ts-0/data/instance:
uuid: "cda8a32a9718472ab94b1634830916b1"
format_stamp: "Formatted at 2026-09-25 15:20:36 on dist-test-slave-4bk3"
server_key: "94f47e407d4143941eb8fc7b78ed827b"
server_key_iv: "cbd497e2c0a5944983af613b94a8113d"
server_key_version: "encryptionkey@0"
I20260925 15:20:36.112084 9556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/ts-0/wal/instance:
uuid: "cda8a32a9718472ab94b1634830916b1"
format_stamp: "Formatted at 2026-09-25 15:20:36 on dist-test-slave-4bk3"
server_key: "94f47e407d4143941eb8fc7b78ed827b"
server_key_iv: "cbd497e2c0a5944983af613b94a8113d"
server_key_version: "encryptionkey@0"
I20260925 15:20:36.113612 9556 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:20:36.114789 9571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:36.115072 9556 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:36.115182 9556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/ts-0/wal
uuid: "cda8a32a9718472ab94b1634830916b1"
format_stamp: "Formatted at 2026-09-25 15:20:36 on dist-test-slave-4bk3"
server_key: "94f47e407d4143941eb8fc7b78ed827b"
server_key_iv: "cbd497e2c0a5944983af613b94a8113d"
server_key_version: "encryptionkey@0"
I20260925 15:20:36.115271 9556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:36.134976 9556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:36.137614 9556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:36.137780 9556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:36.138031 9556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:20:36.138373 9556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:20:36.138417 9556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:36.138440 9556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:20:36.138486 9556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:36.145174 9556 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.193:36441
I20260925 15:20:36.145255 9684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.193:36441 every 8 connection(s)
I20260925 15:20:36.145588 9556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:20:36.147680 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 9556
I20260925 15:20:36.147809 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/ts-0/wal/instance
I20260925 15:20:36.148000 3919 external_mini_cluster.cc:1428] Setting key bede546a576b69be3492d65152c7a851
Sep 25 15:20:36 dist-test-slave-4bk3 krb5kdc[9449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349636, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.193@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.211.194@KRBTEST.COM; defaulting to no policy
I20260925 15:20:36.157063 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:36.146593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:52861 (local address 127.3.211.254:33169)
0925 15:20:36.146675 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:36.146677 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:36.146695 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:20:36.147829 (+ 1134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:36.147831 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:36.147848 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:36.147925 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:36.148790 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:36.149520 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:36.150605 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:36.150939 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:36.151019 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:36.153433 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:36.153435 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:36.153435 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:36.153446 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:36.155195 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:36.155639 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:36.155640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:36.155641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:36.155703 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:36.155948 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:36.155949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:36.155950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:36.156118 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:36.156201 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:36.156821 (+ 620us) server_negotiation.cc:299] Negotiation successful
0925 15:20:36.156937 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 15:20:36.157167 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:36.146379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:33169 (local address 127.3.211.193:52861)
0925 15:20:36.146552 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:36.146598 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:36.147618 (+ 1020us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:36.148016 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:36.148021 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:36.148411 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:36.148695 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:36.148701 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:36.149630 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:36.149631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:36.150515 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:36.150518 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:36.151036 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:36.151037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:36.151147 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:36.151765 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:36.151774 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:36.153321 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:36.155299 (+ 1978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:36.155306 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:36.155308 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:36.155546 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:36.155774 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:36.155774 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:36.155775 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:36.155878 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:36.156219 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:36.156226 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:36.156641 (+ 415us) client_negotiation.cc:769] Sending connection context
0925 15:20:36.156784 (+ 143us) client_negotiation.cc:240] Negotiation successful
0925 15:20:36.156948 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":102,"threads_started":1}
I20260925 15:20:36.157734 9685 heartbeater.cc:347] Connected to a master server at 127.3.211.254:33169
I20260925 15:20:36.157822 9685 heartbeater.cc:464] Registering TS with master...
I20260925 15:20:36.158016 9685 heartbeater.cc:511] Master 127.3.211.254:33169 requested a full tablet report, sending...
I20260925 15:20:36.158560 9494 ts_manager.cc:194] Registered new tserver with Master: cda8a32a9718472ab94b1634830916b1 (127.3.211.193:36441)
I20260925 15:20:36.159217 9494 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.193@KRBTEST.COM'} at 127.3.211.193:52861
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab.
Entry for principal kudu/127.3.211.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab.
I20260925 15:20:36.177616 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.194:0
--local_ip_for_outbound_sockets=127.3.211.194
--webserver_interface=127.3.211.194
--webserver_port=0
--tserver_master_addrs=127.3.211.254:33169
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab
--principal=kudu/127.3.211.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:46761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:20:36.263353 9690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:36.263574 9690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:36.263600 9690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:36.264833 9690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:36.264886 9690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:36.264914 9690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.194
I20260925 15:20:36.266494 9690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:46761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab
--principal=kudu/127.3.211.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.211.194
--webserver_port=0
--tserver_master_addrs=127.3.211.254:33169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.194
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:36.266722 9690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:36.266916 9690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:36.272104 9698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:36.272125 9695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:36.272104 9696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:36.272444 9690 server_base.cc:1034] running on GCE node
I20260925 15:20:36.272816 9690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:36.273036 9690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:36.274188 9690 hybrid_clock.cc:648] HybridClock initialized: now 1790349636274163 us; error 43 us; skew 500 ppm
Sep 25 15:20:36 dist-test-slave-4bk3 krb5kdc[9449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349636, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:36.276844 9690 init.cc:377] Logged in from keytab as kudu/127.3.211.194@KRBTEST.COM (short username kudu)
I20260925 15:20:36.277294 9690 webserver.cc:460] Webserver started at http://127.3.211.194:33787/ using document root <none> and password file <none>
I20260925 15:20:36.277550 9690 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:36.277621 9690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:36.277777 9690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:36.278879 9690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/ts-1/data/instance:
uuid: "09f50d165f5a428a8af330304ec2b275"
format_stamp: "Formatted at 2026-09-25 15:20:36 on dist-test-slave-4bk3"
server_key: "648ffecc5b1974e9f4c8df39c9b42a59"
server_key_iv: "845aa752bd258e0a232cb5be07021dd8"
server_key_version: "encryptionkey@0"
I20260925 15:20:36.279253 9690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/ts-1/wal/instance:
uuid: "09f50d165f5a428a8af330304ec2b275"
format_stamp: "Formatted at 2026-09-25 15:20:36 on dist-test-slave-4bk3"
server_key: "648ffecc5b1974e9f4c8df39c9b42a59"
server_key_iv: "845aa752bd258e0a232cb5be07021dd8"
server_key_version: "encryptionkey@0"
I20260925 15:20:36.280723 9690 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:20:36.281529 9705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:36.281724 9690 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:20:36.281829 9690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/ts-1/wal
uuid: "09f50d165f5a428a8af330304ec2b275"
format_stamp: "Formatted at 2026-09-25 15:20:36 on dist-test-slave-4bk3"
server_key: "648ffecc5b1974e9f4c8df39c9b42a59"
server_key_iv: "845aa752bd258e0a232cb5be07021dd8"
server_key_version: "encryptionkey@0"
I20260925 15:20:36.281903 9690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:36.301553 9690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:36.304008 9690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:36.304184 9690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:36.304486 9690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:20:36.304807 9690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:20:36.304859 9690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:36.304901 9690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:20:36.304927 9690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:36.310910 9690 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.194:39913
I20260925 15:20:36.311002 9818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.194:39913 every 8 connection(s)
I20260925 15:20:36.311332 9690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
I20260925 15:20:36.313273 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 9690
I20260925 15:20:36.313400 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/ts-1/wal/instance
I20260925 15:20:36.313598 3919 external_mini_cluster.cc:1428] Setting key 4ea5d4e671335ec3dee2f513e39e0073
Sep 25 15:20:36 dist-test-slave-4bk3 krb5kdc[9449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349636, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.194@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.211.195@KRBTEST.COM; defaulting to no policy
I20260925 15:20:36.323212 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:36.312208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:60045 (local address 127.3.211.254:33169)
0925 15:20:36.312346 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:36.312348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:36.312513 (+ 165us) server_negotiation.cc:407] Connection header received
0925 15:20:36.313797 (+ 1284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:36.313799 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:36.313814 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:36.313878 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:36.314759 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:36.315256 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:36.316692 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:36.317014 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:36.317084 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:36.319755 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:36.319757 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:36.319757 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:36.319770 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:36.321684 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:36.322208 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:36.322209 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:36.322209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:36.322273 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:36.322593 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:36.322594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:36.322594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:36.322729 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:36.322795 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:36.323062 (+ 267us) server_negotiation.cc:299] Negotiation successful
0925 15:20:36.323143 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 15:20:36.323436 9821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:36.312092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:33169 (local address 127.3.211.194:60045)
0925 15:20:36.312383 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:36.312427 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:36.313638 (+ 1211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:36.313968 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:36.313973 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:36.314344 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:36.314648 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:36.314654 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:36.315373 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:36.315375 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:36.316598 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:36.316601 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:36.317121 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:36.317122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:36.317239 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:36.317828 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:36.317836 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:36.319606 (+ 1770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:36.321773 (+ 2167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:36.321783 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:36.321786 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:36.322124 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:36.322341 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:36.322341 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:36.322342 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:36.322506 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:36.322799 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:36.322804 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:36.322937 (+ 133us) client_negotiation.cc:769] Sending connection context
0925 15:20:36.323030 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 15:20:36.323166 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":86,"threads_started":1}
I20260925 15:20:36.323983 9819 heartbeater.cc:347] Connected to a master server at 127.3.211.254:33169
I20260925 15:20:36.324072 9819 heartbeater.cc:464] Registering TS with master...
I20260925 15:20:36.324268 9819 heartbeater.cc:511] Master 127.3.211.254:33169 requested a full tablet report, sending...
I20260925 15:20:36.324776 9494 ts_manager.cc:194] Registered new tserver with Master: 09f50d165f5a428a8af330304ec2b275 (127.3.211.194:39913)
I20260925 15:20:36.325233 9494 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.194@KRBTEST.COM'} at 127.3.211.194:60045
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab.
Entry for principal kudu/127.3.211.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab.
I20260925 15:20:36.343339 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.195:0
--local_ip_for_outbound_sockets=127.3.211.195
--webserver_interface=127.3.211.195
--webserver_port=0
--tserver_master_addrs=127.3.211.254:33169
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab
--principal=kudu/127.3.211.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:46761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:20:36.429550 9824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:36.429821 9824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:36.429849 9824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:36.431313 9824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:36.431370 9824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:36.431402 9824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.195
I20260925 15:20:36.433398 9824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:46761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab
--principal=kudu/127.3.211.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.211.195
--webserver_port=0
--tserver_master_addrs=127.3.211.254:33169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.195
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:36.433662 9824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:36.433862 9824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:36.439371 9829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:36.439356 9830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:36.439441 9824 server_base.cc:1034] running on GCE node
W20260925 15:20:36.439463 9832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:36.439877 9824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:36.440129 9824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:36.441310 9824 hybrid_clock.cc:648] HybridClock initialized: now 1790349636441269 us; error 52 us; skew 500 ppm
Sep 25 15:20:36 dist-test-slave-4bk3 krb5kdc[9449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349636, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:36.444283 9824 init.cc:377] Logged in from keytab as kudu/127.3.211.195@KRBTEST.COM (short username kudu)
I20260925 15:20:36.444905 9824 webserver.cc:460] Webserver started at http://127.3.211.195:38397/ using document root <none> and password file <none>
I20260925 15:20:36.445154 9824 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:36.445202 9824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:36.445366 9824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:36.446501 9824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/ts-2/data/instance:
uuid: "0e1bb8188e754cc2ac724ffac1bdc672"
format_stamp: "Formatted at 2026-09-25 15:20:36 on dist-test-slave-4bk3"
server_key: "f5e143245d8d3024d8f64dcb455e705c"
server_key_iv: "8baa32f7c54c082d63cd591cb880523c"
server_key_version: "encryptionkey@0"
I20260925 15:20:36.446858 9824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/ts-2/wal/instance:
uuid: "0e1bb8188e754cc2ac724ffac1bdc672"
format_stamp: "Formatted at 2026-09-25 15:20:36 on dist-test-slave-4bk3"
server_key: "f5e143245d8d3024d8f64dcb455e705c"
server_key_iv: "8baa32f7c54c082d63cd591cb880523c"
server_key_version: "encryptionkey@0"
I20260925 15:20:36.448395 9824 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:20:36.449306 9839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:36.449523 9824 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:36.449633 9824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/ts-2/wal
uuid: "0e1bb8188e754cc2ac724ffac1bdc672"
format_stamp: "Formatted at 2026-09-25 15:20:36 on dist-test-slave-4bk3"
server_key: "f5e143245d8d3024d8f64dcb455e705c"
server_key_iv: "8baa32f7c54c082d63cd591cb880523c"
server_key_version: "encryptionkey@0"
I20260925 15:20:36.449729 9824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:36.470122 9824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:36.472620 9824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:36.472792 9824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:36.473048 9824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:20:36.473368 9824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:20:36.473423 9824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:36.473470 9824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:20:36.473505 9824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:36.480187 9824 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.195:41389
I20260925 15:20:36.480230 9952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.195:41389 every 8 connection(s)
I20260925 15:20:36.480619 9824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
Sep 25 15:20:36 dist-test-slave-4bk3 krb5kdc[9449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349636, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.195@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:20:36.488929 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 9824
I20260925 15:20:36.489038 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/minicluster-data/ts-2/wal/instance
I20260925 15:20:36.489214 3919 external_mini_cluster.cc:1428] Setting key dfcb690e77a71a0ef2dc67e16f745a76
I20260925 15:20:36.490911 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:36.481365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:45665 (local address 127.3.211.254:33169)
0925 15:20:36.481424 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:36.481426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:36.481644 (+ 218us) server_negotiation.cc:407] Connection header received
0925 15:20:36.482648 (+ 1004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:36.482650 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:36.482664 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:36.482715 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:36.483418 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:36.483937 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:36.485069 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:36.485290 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:36.485379 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:36.487799 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:36.487800 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:36.487801 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:36.487808 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:36.489403 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:36.489954 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:36.489955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:36.489955 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:36.490034 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:36.490284 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:36.490285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:36.490285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:36.490406 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:36.490461 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:36.490782 (+ 321us) server_negotiation.cc:299] Negotiation successful
0925 15:20:36.490846 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 15:20:36.491197 9955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:36.481363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:33169 (local address 127.3.211.195:45665)
0925 15:20:36.481525 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:36.481561 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:36.482520 (+ 959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:36.482796 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:36.482799 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:36.483089 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:36.483317 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:36.483322 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:36.484033 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:36.484035 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:36.484921 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:36.484923 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:36.485392 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:36.485393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:36.485521 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:36.486053 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:36.486062 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:36.487694 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:36.489516 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:36.489527 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:36.489529 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:36.489877 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:36.490101 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:36.490102 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:36.490102 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:36.490228 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:36.490484 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:36.490490 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:36.490657 (+ 167us) client_negotiation.cc:769] Sending connection context
0925 15:20:36.490759 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 15:20:36.490895 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":77,"threads_started":1}
I20260925 15:20:36.491753 9953 heartbeater.cc:347] Connected to a master server at 127.3.211.254:33169
I20260925 15:20:36.491887 9953 heartbeater.cc:464] Registering TS with master...
I20260925 15:20:36.492120 9953 heartbeater.cc:511] Master 127.3.211.254:33169 requested a full tablet report, sending...
I20260925 15:20:36.492581 9494 ts_manager.cc:194] Registered new tserver with Master: 0e1bb8188e754cc2ac724ffac1bdc672 (127.3.211.195:41389)
I20260925 15:20:36.492990 9494 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.195@KRBTEST.COM'} at 127.3.211.195:45665
I20260925 15:20:36.501573 3919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/krb5kdc/test-admin.keytab.
Sep 25 15:20:36 dist-test-slave-4bk3 krb5kdc[9449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:36.518450 3919 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 15:20:36.520223 9960 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:36.519435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60164 (local address 127.3.211.193:36441)
0925 15:20:36.519614 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:36.519620 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:36.519630 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:20:36.519656 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:36.519658 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:36.519682 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:36.519763 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:36.520136 (+ 373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60164: BlockingRecv error: recv got EOF from 127.0.0.1:60164 (error 108)
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":79,"threads_started":1}
W20260925 15:20:36.520356 3919 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/krb5kdc/krb5cc' of type FILE
Sep 25 15:20:36 dist-test-slave-4bk3 krb5kdc[9449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:36.521420 3919 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 15:20:36 dist-test-slave-4bk3 krb5kdc[9449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:36.522336 3919 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 15:20:36 dist-test-slave-4bk3 krb5kdc[9449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.211.193@KRBTEST.COM
I20260925 15:20:36.531363 9960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:36.522963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60176 (local address 127.3.211.193:36441)
0925 15:20:36.523017 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:36.523019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:36.523030 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:20:36.523102 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:36.523103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:36.523113 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:36.523162 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:36.523660 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:36.524535 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:36.525427 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:36.525797 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:36.525868 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:36.528209 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:36.528219 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:36.528220 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:36.528227 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:36.530074 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:36.530469 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:36.530472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:36.530473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:36.530535 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:36.530718 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:36.530719 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:36.530719 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:36.530864 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:36.530977 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:36.531109 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 15:20:36.531179 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20260925 15:20:36.533484 9960 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:36.532827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60178 (local address 127.3.211.193:36441)
0925 15:20:36.532888 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:36.532890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:36.532927 (+ 37us) server_negotiation.cc:407] Connection header received
0925 15:20:36.532978 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:36.532979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:36.532989 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:36.533036 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:36.533437 (+ 401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60178: BlockingRecv error: recv got EOF from 127.0.0.1:60178 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260925 15:20:36.533602 3919 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/krb5kdc/krb5cc' of type FILE
Sep 25 15:20:36 dist-test-slave-4bk3 krb5kdc[9449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:36.534677 3919 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 15:20:36 dist-test-slave-4bk3 krb5kdc[9449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:36.535894 3919 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 15:20:36 dist-test-slave-4bk3 krb5kdc[9449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.211.193@KRBTEST.COM
I20260925 15:20:36.543478 9960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:36.536507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60190 (local address 127.3.211.193:36441)
0925 15:20:36.536561 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:36.536563 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:36.536661 (+ 98us) server_negotiation.cc:407] Connection header received
0925 15:20:36.536716 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:36.536717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:36.536727 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:36.536768 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:36.537134 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:36.537616 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:36.538324 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:36.538541 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:36.538605 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:36.540881 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:36.540883 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:36.540883 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:36.540890 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:36.542191 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:36.542543 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:36.542545 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:36.542545 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:36.542599 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:36.542897 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:36.542898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:36.542898 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:36.543013 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:36.543107 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:36.543279 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 15:20:36.543353 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20260925 15:20:36.545193 9960 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:36.544647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60202 (local address 127.3.211.193:36441)
0925 15:20:36.544707 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:36.544709 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:36.544717 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:20:36.544793 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:36.544794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:36.544802 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:36.544852 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:36.545146 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60202: BlockingRecv error: recv got EOF from 127.0.0.1:60202 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260925 15:20:36.545360 3919 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790349610595785-3919-0/krb5kdc/krb5cc)
Sep 25 15:20:36 dist-test-slave-4bk3 krb5kdc[9449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:36.546428 3919 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 15:20:36 dist-test-slave-4bk3 krb5kdc[9449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:36.547369 3919 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 15:20:36 dist-test-slave-4bk3 krb5kdc[9449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.211.193@KRBTEST.COM
I20260925 15:20:36.554984 9960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:36.547925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60210 (local address 127.3.211.193:36441)
0925 15:20:36.547980 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:36.547982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:36.548050 (+ 68us) server_negotiation.cc:407] Connection header received
0925 15:20:36.548138 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:36.548139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:36.548148 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:36.548212 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:36.548691 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:36.549155 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:36.549842 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:36.550027 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:36.550108 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:36.552206 (+ 2098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:36.552208 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:36.552208 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:36.552215 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:36.553717 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:36.554062 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:36.554063 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:36.554063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:36.554119 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:36.554361 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:36.554362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:36.554362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:36.554473 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:36.554574 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:36.554836 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 15:20:36.554906 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 15:20:36.555773 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 9556
I20260925 15:20:36.560516 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 9690
I20260925 15:20:36.564898 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 9824
I20260925 15:20:36.569242 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 9462
2026-09-25T15:20:36Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1963 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:20:36 dist-test-slave-4bk3 krb5kdc[9973](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:20:36 dist-test-slave-4bk3 krb5kdc[9973](info): set up 2 sockets
Sep 25 15:20:36 dist-test-slave-4bk3 krb5kdc[9973](info): commencing operation
krb5kdc: starting...
W20260925 15:20:37.620954 3919 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Sep 25 15:20:37 dist-test-slave-4bk3 krb5kdc[9973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:20:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:20:37Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.211.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.211.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.211.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/oryx_127.3.211.254.keytab.
Entry for principal oryx/127.3.211.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/oryx_127.3.211.254.keytab.
W20260925 15:20:38.445401 3919 mini_kdc.cc:262] Time spent creating service keytab for oryx/127.3.211.254: real 0.721s user 0.000s sys 0.003s
I20260925 15:20:38.445835 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:38373
--webserver_interface=127.3.211.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/oryx_127.3.211.254.keytab
--principal=oryx/127.3.211.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:42075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.211.254:38373
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:20:38.537964 9986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:38.538285 9986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:38.538316 9986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:38.539925 9986 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:20:38.539975 9986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:38.539991 9986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:38.540004 9986 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:20:38.540017 9986 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:20:38.542949 9986 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:42075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.211.254:38373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/oryx_127.3.211.254.keytab
--principal=oryx/127.3.211.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:38373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.211.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:38.543236 9986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:38.543439 9986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:38.548642 9992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:38.548657 9991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:38.548673 9994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:38.548995 9986 server_base.cc:1034] running on GCE node
I20260925 15:20:38.549300 9986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:38.549652 9986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:38.550835 9986 hybrid_clock.cc:648] HybridClock initialized: now 1790349638550806 us; error 48 us; skew 500 ppm
Sep 25 15:20:38 dist-test-slave-4bk3 krb5kdc[9973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349638, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.211.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:38.559315 9986 init.cc:377] Logged in from keytab as oryx/127.3.211.254@KRBTEST.COM (short username oryx)
I20260925 15:20:38.559895 9986 webserver.cc:460] Webserver started at http://127.3.211.254:43187/ using document root <none> and password file <none>
I20260925 15:20:38.560159 9986 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:38.560207 9986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:38.560401 9986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:38.561568 9986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-0/data/instance:
uuid: "b3860a4600f94b6fa48e2ac45ace80e8"
format_stamp: "Formatted at 2026-09-25 15:20:38 on dist-test-slave-4bk3"
server_key: "334d99ceab055a7dda2bf2e034b92a02"
server_key_iv: "6156d250d02611ac1aabe3d18b12216a"
server_key_version: "encryptionkey@0"
I20260925 15:20:38.561965 9986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-0/wal/instance:
uuid: "b3860a4600f94b6fa48e2ac45ace80e8"
format_stamp: "Formatted at 2026-09-25 15:20:38 on dist-test-slave-4bk3"
server_key: "334d99ceab055a7dda2bf2e034b92a02"
server_key_iv: "6156d250d02611ac1aabe3d18b12216a"
server_key_version: "encryptionkey@0"
I20260925 15:20:38.563453 9986 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:20:38.564447 10001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:38.564639 9986 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:38.564735 9986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-0/wal
uuid: "b3860a4600f94b6fa48e2ac45ace80e8"
format_stamp: "Formatted at 2026-09-25 15:20:38 on dist-test-slave-4bk3"
server_key: "334d99ceab055a7dda2bf2e034b92a02"
server_key_iv: "6156d250d02611ac1aabe3d18b12216a"
server_key_version: "encryptionkey@0"
I20260925 15:20:38.564800 9986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:38.580011 9986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:38.582470 9986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:38.582595 9986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:38.587123 9986 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.254:38373
I20260925 15:20:38.587175 10053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.254:38373 every 8 connection(s)
I20260925 15:20:38.587626 9986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
I20260925 15:20:38.588809 10054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:38.591324 10054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8: Bootstrap starting.
I20260925 15:20:38.591835 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 9986
I20260925 15:20:38.591946 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-0/wal/instance
I20260925 15:20:38.592111 10054 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:38.592144 3919 external_mini_cluster.cc:1428] Setting key 1967b3e4812f7057f001d8ca1e930028
I20260925 15:20:38.592465 10054 log.cc:824] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8: Log is configured to *not* fsync() on all Append() calls
I20260925 15:20:38.593377 10054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8: No bootstrap required, opened a new log
I20260925 15:20:38.595886 10054 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38373 } }
I20260925 15:20:38.596097 10054 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:38.596141 10054 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3860a4600f94b6fa48e2ac45ace80e8, State: Initialized, Role: FOLLOWER
I20260925 15:20:38.596283 10054 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38373 } }
I20260925 15:20:38.596410 10054 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:20:38.596473 10054 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:20:38.596524 10054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8 [term 0 FOLLOWER]: Advancing to term 1
Sep 25 15:20:38 dist-test-slave-4bk3 krb5kdc[9973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM
I20260925 15:20:38.603668 10058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:38.592810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47566 (local address 127.3.211.254:38373)
0925 15:20:38.592998 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:38.593008 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:38.593044 (+ 36us) server_negotiation.cc:407] Connection header received
0925 15:20:38.593696 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:38.593705 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:38.593795 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:38.593962 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:38.594528 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:38.595994 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:38.597028 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:38.597523 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:38.597616 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:38.599978 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:38.599982 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:38.599983 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:38.599996 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:38.601861 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:38.602327 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:38.602331 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:38.602331 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:38.602429 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:38.602739 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:38.602740 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:38.602740 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:38.603019 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:38.603137 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:38.603255 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 15:20:38.603374 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":104,"threads_started":1}
I20260925 15:20:38.652801 10054 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38373 } }
I20260925 15:20:38.653012 10054 leader_election.cc:302] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3860a4600f94b6fa48e2ac45ace80e8; no voters:
I20260925 15:20:38.653294 10054 leader_election.cc:288] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:20:38.653401 10059 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:20:38.653610 10054 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:20:38.653635 10059 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8 [term 1 LEADER]: Becoming Leader. State: Replica: b3860a4600f94b6fa48e2ac45ace80e8, State: Running, Role: LEADER
I20260925 15:20:38.653877 10059 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38373 } }
I20260925 15:20:38.654357 10060 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38373 } } }
I20260925 15:20:38.654464 10060 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8 [sys.catalog]: This master's current role is: LEADER
I20260925 15:20:38.654609 10061 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3860a4600f94b6fa48e2ac45ace80e8. Latest consensus state: current_term: 1 leader_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38373 } } }
I20260925 15:20:38.654706 10061 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8 [sys.catalog]: This master's current role is: LEADER
I20260925 15:20:38.655133 10068 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:20:38.656780 10068 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:20:38.658782 10068 catalog_manager.cc:1347] Generated new cluster ID: f33e811646eb40de953a49f346770393
I20260925 15:20:38.658855 10068 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:20:38.668452 10068 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:20:38.669112 10068 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:20:38.675046 10068 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8: Generated new TSK 0
I20260925 15:20:38.675278 10068 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.211.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.211.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.211.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/oryx_127.3.211.193.keytab.
Entry for principal oryx/127.3.211.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/oryx_127.3.211.193.keytab.
W20260925 15:20:39.720801 3919 mini_kdc.cc:262] Time spent creating service keytab for oryx/127.3.211.193: real 1.041s user 0.000s sys 0.003s
I20260925 15:20:39.721129 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.193:0
--local_ip_for_outbound_sockets=127.3.211.193
--webserver_interface=127.3.211.193
--webserver_port=0
--tserver_master_addrs=127.3.211.254:38373
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/oryx_127.3.211.193.keytab
--principal=oryx/127.3.211.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:42075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:20:39.805915 10080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:39.806170 10080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:39.806212 10080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:39.807401 10080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:39.807485 10080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:39.807531 10080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.193
I20260925 15:20:39.809087 10080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:42075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/oryx_127.3.211.193.keytab
--principal=oryx/127.3.211.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.211.193
--webserver_port=0
--tserver_master_addrs=127.3.211.254:38373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.193
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:39.809374 10080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:39.809618 10080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:39.814934 10086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:39.814807 10085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:39.815075 10080 server_base.cc:1034] running on GCE node
W20260925 15:20:39.814823 10088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:39.815312 10080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:39.815526 10080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:39.816659 10080 hybrid_clock.cc:648] HybridClock initialized: now 1790349639816646 us; error 31 us; skew 500 ppm
Sep 25 15:20:39 dist-test-slave-4bk3 krb5kdc[9973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349639, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.211.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:39.819090 10080 init.cc:377] Logged in from keytab as oryx/127.3.211.193@KRBTEST.COM (short username oryx)
I20260925 15:20:39.819588 10080 webserver.cc:460] Webserver started at http://127.3.211.193:41623/ using document root <none> and password file <none>
I20260925 15:20:39.819836 10080 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:39.819904 10080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:39.820047 10080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:39.821195 10080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-0/data/instance:
uuid: "bf6c0d3ea9e342d6960a6f8a5d1e7e26"
format_stamp: "Formatted at 2026-09-25 15:20:39 on dist-test-slave-4bk3"
server_key: "3f20fad442104bdcc87156ef5e45468f"
server_key_iv: "cea7526e4382ddf1b0879d046696c9f5"
server_key_version: "encryptionkey@0"
I20260925 15:20:39.821571 10080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-0/wal/instance:
uuid: "bf6c0d3ea9e342d6960a6f8a5d1e7e26"
format_stamp: "Formatted at 2026-09-25 15:20:39 on dist-test-slave-4bk3"
server_key: "3f20fad442104bdcc87156ef5e45468f"
server_key_iv: "cea7526e4382ddf1b0879d046696c9f5"
server_key_version: "encryptionkey@0"
I20260925 15:20:39.823014 10080 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 15:20:39.823902 10095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:39.824096 10080 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:20:39.824213 10080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-0/wal
uuid: "bf6c0d3ea9e342d6960a6f8a5d1e7e26"
format_stamp: "Formatted at 2026-09-25 15:20:39 on dist-test-slave-4bk3"
server_key: "3f20fad442104bdcc87156ef5e45468f"
server_key_iv: "cea7526e4382ddf1b0879d046696c9f5"
server_key_version: "encryptionkey@0"
I20260925 15:20:39.824295 10080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:39.858804 10080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:39.861115 10080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:39.861389 10080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:39.861665 10080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:20:39.861994 10080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:20:39.862047 10080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:39.862089 10080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:20:39.862131 10080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:39.868150 10080 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.193:32847
I20260925 15:20:39.868199 10208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.193:32847 every 8 connection(s)
I20260925 15:20:39.868646 10080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
Sep 25 15:20:39 dist-test-slave-4bk3 krb5kdc[9973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349639, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.211.193@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM
I20260925 15:20:39.876935 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 10080
I20260925 15:20:39.877043 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-0/wal/instance
I20260925 15:20:39.877257 3919 external_mini_cluster.cc:1428] Setting key 150ad0fe683a61f6e25b7cc5746f6ca5
I20260925 15:20:39.879280 10212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:39.869391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38373 (local address 127.3.211.193:54825)
0925 15:20:39.869594 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:39.869646 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:39.870197 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:39.870479 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:39.870483 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:39.871087 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:39.871314 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:39.871319 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:39.872075 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:39.872077 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:39.872953 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:39.872956 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:39.873387 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:39.873388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:39.873519 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:39.874105 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:39.874114 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:39.875636 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:39.877512 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:39.877524 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:39.877527 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:39.877876 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:39.878108 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:39.878109 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:39.878109 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:39.878244 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:39.878532 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:39.878537 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:39.878665 (+ 128us) client_negotiation.cc:769] Sending connection context
0925 15:20:39.878936 (+ 271us) client_negotiation.cc:240] Negotiation successful
0925 15:20:39.879096 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":88,"threads_started":1}
I20260925 15:20:39.879271 10211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:39.869402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:54825 (local address 127.3.211.254:38373)
0925 15:20:39.869580 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:39.869583 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:39.869751 (+ 168us) server_negotiation.cc:407] Connection header received
0925 15:20:39.870302 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:39.870303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:39.870329 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:39.870404 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:39.871420 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:39.871962 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:39.873054 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:39.873284 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:39.873373 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:39.875752 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:39.875753 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:39.875753 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:39.875761 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:39.877418 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:39.877960 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:39.877962 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:39.877962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:39.878033 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:39.878307 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:39.878309 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:39.878309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:39.878459 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:39.878536 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:39.879001 (+ 465us) server_negotiation.cc:299] Negotiation successful
0925 15:20:39.879118 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":68,"threads_started":1}
I20260925 15:20:39.880112 10209 heartbeater.cc:347] Connected to a master server at 127.3.211.254:38373
I20260925 15:20:39.880247 10209 heartbeater.cc:464] Registering TS with master...
I20260925 15:20:39.880468 10209 heartbeater.cc:511] Master 127.3.211.254:38373 requested a full tablet report, sending...
I20260925 15:20:39.881114 10018 ts_manager.cc:194] Registered new tserver with Master: bf6c0d3ea9e342d6960a6f8a5d1e7e26 (127.3.211.193:32847)
I20260925 15:20:39.882051 10018 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.211.193@KRBTEST.COM'} at 127.3.211.193:54825
WARNING: no policy specified for oryx/127.3.211.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.211.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.211.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/oryx_127.3.211.194.keytab.
Entry for principal oryx/127.3.211.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/oryx_127.3.211.194.keytab.
I20260925 15:20:39.905098 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.194:0
--local_ip_for_outbound_sockets=127.3.211.194
--webserver_interface=127.3.211.194
--webserver_port=0
--tserver_master_addrs=127.3.211.254:38373
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/oryx_127.3.211.194.keytab
--principal=oryx/127.3.211.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:42075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:20:39.988677 10215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:39.988910 10215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:39.988937 10215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:39.990242 10215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:39.990295 10215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:39.990339 10215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.194
I20260925 15:20:39.991930 10215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:42075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/oryx_127.3.211.194.keytab
--principal=oryx/127.3.211.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.211.194
--webserver_port=0
--tserver_master_addrs=127.3.211.254:38373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.194
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:39.992218 10215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:39.992489 10215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:39.997838 10221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:39.997838 10223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:39.997877 10215 server_base.cc:1034] running on GCE node
W20260925 15:20:39.997841 10220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:39.998332 10215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:39.998620 10215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:39.999806 10215 hybrid_clock.cc:648] HybridClock initialized: now 1790349639999796 us; error 39 us; skew 500 ppm
Sep 25 15:20:39 dist-test-slave-4bk3 krb5kdc[9973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349639, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.211.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:40.002637 10215 init.cc:377] Logged in from keytab as oryx/127.3.211.194@KRBTEST.COM (short username oryx)
I20260925 15:20:40.003129 10215 webserver.cc:460] Webserver started at http://127.3.211.194:46841/ using document root <none> and password file <none>
I20260925 15:20:40.003337 10215 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:40.003381 10215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:40.003476 10215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:40.004586 10215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-1/data/instance:
uuid: "7921060cf7c04ccf844a869c26b58c4a"
format_stamp: "Formatted at 2026-09-25 15:20:39 on dist-test-slave-4bk3"
server_key: "c2dd285cb4fcdd087bc663304a7cb4ca"
server_key_iv: "dddf2e047dcf47c9e97a37ee60135893"
server_key_version: "encryptionkey@0"
I20260925 15:20:40.004894 10215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-1/wal/instance:
uuid: "7921060cf7c04ccf844a869c26b58c4a"
format_stamp: "Formatted at 2026-09-25 15:20:39 on dist-test-slave-4bk3"
server_key: "c2dd285cb4fcdd087bc663304a7cb4ca"
server_key_iv: "dddf2e047dcf47c9e97a37ee60135893"
server_key_version: "encryptionkey@0"
I20260925 15:20:40.006289 10215 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:40.007550 10230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:40.007735 10215 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:40.007825 10215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-1/wal
uuid: "7921060cf7c04ccf844a869c26b58c4a"
format_stamp: "Formatted at 2026-09-25 15:20:39 on dist-test-slave-4bk3"
server_key: "c2dd285cb4fcdd087bc663304a7cb4ca"
server_key_iv: "dddf2e047dcf47c9e97a37ee60135893"
server_key_version: "encryptionkey@0"
I20260925 15:20:40.007889 10215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:40.017601 10215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:40.019609 10215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:40.019726 10215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:40.019932 10215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:20:40.020301 10215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:20:40.020402 10215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:40.020455 10215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:20:40.020484 10215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:40.026269 10215 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.194:44803
I20260925 15:20:40.026355 10343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.194:44803 every 8 connection(s)
I20260925 15:20:40.026686 10215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
I20260925 15:20:40.030552 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 10215
I20260925 15:20:40.030665 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-1/wal/instance
I20260925 15:20:40.030871 3919 external_mini_cluster.cc:1428] Setting key e8f702769ed6f72251ec491a60569ee0
Sep 25 15:20:40 dist-test-slave-4bk3 krb5kdc[9973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349639, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.211.194@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM
I20260925 15:20:40.039425 10211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:40.028029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:55153 (local address 127.3.211.254:38373)
0925 15:20:40.028141 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:40.028144 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:40.028598 (+ 454us) server_negotiation.cc:407] Connection header received
0925 15:20:40.029174 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:40.029175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:40.029192 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:40.029253 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:40.030377 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.031163 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.032439 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.032658 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.032753 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:40.035483 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:40.035485 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:40.035485 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:40.035493 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:40.037335 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:40.037806 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:40.037808 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:40.037808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:40.037881 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:40.038129 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:40.038130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:40.038131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:40.038297 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:40.038485 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:40.039159 (+ 674us) server_negotiation.cc:299] Negotiation successful
0925 15:20:40.039275 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 15:20:40.039521 10346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:40.027959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38373 (local address 127.3.211.194:55153)
0925 15:20:40.028361 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:40.028431 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:40.029055 (+ 624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:40.029329 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:40.029333 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:40.029970 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:40.030181 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:40.030186 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.031291 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.031293 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:40.032264 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:40.032267 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.032768 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.032769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:40.032910 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:40.033517 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:40.033526 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:40.035369 (+ 1843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:40.037441 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:40.037449 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:40.037451 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:40.037710 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:40.037956 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:40.037957 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:40.037957 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:40.038062 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:40.038460 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:40.038465 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:40.038607 (+ 142us) client_negotiation.cc:769] Sending connection context
0925 15:20:40.038893 (+ 286us) client_negotiation.cc:240] Negotiation successful
0925 15:20:40.039287 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":188,"threads_started":1}
I20260925 15:20:40.040078 10344 heartbeater.cc:347] Connected to a master server at 127.3.211.254:38373
I20260925 15:20:40.040189 10344 heartbeater.cc:464] Registering TS with master...
I20260925 15:20:40.040441 10344 heartbeater.cc:511] Master 127.3.211.254:38373 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.3.211.195@KRBTEST.COM; defaulting to no policy
I20260925 15:20:40.040904 10018 ts_manager.cc:194] Registered new tserver with Master: 7921060cf7c04ccf844a869c26b58c4a (127.3.211.194:44803)
I20260925 15:20:40.041314 10018 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.211.194@KRBTEST.COM'} at 127.3.211.194:55153
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.211.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.211.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/oryx_127.3.211.195.keytab.
Entry for principal oryx/127.3.211.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/oryx_127.3.211.195.keytab.
I20260925 15:20:40.059837 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.195:0
--local_ip_for_outbound_sockets=127.3.211.195
--webserver_interface=127.3.211.195
--webserver_port=0
--tserver_master_addrs=127.3.211.254:38373
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/oryx_127.3.211.195.keytab
--principal=oryx/127.3.211.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:42075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:20:40.146432 10349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:40.146682 10349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:40.146711 10349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:40.148536 10349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:40.148660 10349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:40.148710 10349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.195
I20260925 15:20:40.151513 10349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:42075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/oryx_127.3.211.195.keytab
--principal=oryx/127.3.211.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.211.195
--webserver_port=0
--tserver_master_addrs=127.3.211.254:38373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.195
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:40.151875 10349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:40.152151 10349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:40.157946 10354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:40.157922 10355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:40.158252 10349 server_base.cc:1034] running on GCE node
W20260925 15:20:40.157960 10357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:40.158632 10349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:40.158923 10349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:40.160099 10349 hybrid_clock.cc:648] HybridClock initialized: now 1790349640160097 us; error 43 us; skew 500 ppm
Sep 25 15:20:40 dist-test-slave-4bk3 krb5kdc[9973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349640, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.211.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:40.162878 10349 init.cc:377] Logged in from keytab as oryx/127.3.211.195@KRBTEST.COM (short username oryx)
I20260925 15:20:40.163352 10349 webserver.cc:460] Webserver started at http://127.3.211.195:41227/ using document root <none> and password file <none>
I20260925 15:20:40.163592 10349 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:40.163656 10349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:40.163805 10349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:40.165009 10349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-2/data/instance:
uuid: "fbc183e336c74b8a8d68f4624d9b375b"
format_stamp: "Formatted at 2026-09-25 15:20:40 on dist-test-slave-4bk3"
server_key: "06cd2690c772a1743ce2d60cf6bff7f6"
server_key_iv: "9a0115d5a88acfac4a5afdecd2eba7cf"
server_key_version: "encryptionkey@0"
I20260925 15:20:40.165375 10349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-2/wal/instance:
uuid: "fbc183e336c74b8a8d68f4624d9b375b"
format_stamp: "Formatted at 2026-09-25 15:20:40 on dist-test-slave-4bk3"
server_key: "06cd2690c772a1743ce2d60cf6bff7f6"
server_key_iv: "9a0115d5a88acfac4a5afdecd2eba7cf"
server_key_version: "encryptionkey@0"
I20260925 15:20:40.166778 10349 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:20:40.168028 10364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:40.168316 10349 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:40.168468 10349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-2/wal
uuid: "fbc183e336c74b8a8d68f4624d9b375b"
format_stamp: "Formatted at 2026-09-25 15:20:40 on dist-test-slave-4bk3"
server_key: "06cd2690c772a1743ce2d60cf6bff7f6"
server_key_iv: "9a0115d5a88acfac4a5afdecd2eba7cf"
server_key_version: "encryptionkey@0"
I20260925 15:20:40.168560 10349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:40.189181 10349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:40.191593 10349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:40.191766 10349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:40.192008 10349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:20:40.192374 10349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:20:40.192430 10349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:40.192469 10349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:20:40.192497 10349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:40.198350 10349 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.195:34379
I20260925 15:20:40.198418 10477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.195:34379 every 8 connection(s)
I20260925 15:20:40.198896 10349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
Sep 25 15:20:40 dist-test-slave-4bk3 krb5kdc[9973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349640, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.211.195@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM
I20260925 15:20:40.205819 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 10349
I20260925 15:20:40.205945 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-2/wal/instance
I20260925 15:20:40.206225 3919 external_mini_cluster.cc:1428] Setting key 2ce70cbaed588b5e16c8fc26dc95dddc
I20260925 15:20:40.209613 10211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:40.199858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:55983 (local address 127.3.211.254:38373)
0925 15:20:40.199962 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:40.199965 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:40.200133 (+ 168us) server_negotiation.cc:407] Connection header received
0925 15:20:40.200738 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:40.200739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:40.200752 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:40.200809 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:40.201820 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.202326 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.203378 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.203608 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.203695 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:40.206121 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:40.206127 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:40.206128 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:40.206136 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:40.208074 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:40.208632 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:40.208633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:40.208633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:40.208700 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:40.208994 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:40.208995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:40.208995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:40.209110 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:40.209162 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:40.209472 (+ 310us) server_negotiation.cc:299] Negotiation successful
0925 15:20:40.209546 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 15:20:40.209789 10480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:40.199782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38373 (local address 127.3.211.195:55983)
0925 15:20:40.199992 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:40.200037 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:40.200629 (+ 592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:40.200890 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:40.200894 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:40.201498 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:40.201709 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:40.201714 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.202437 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.202438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:40.203280 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:40.203282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.203706 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.203707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:40.203829 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:40.204426 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:40.204436 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:40.206016 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:40.208161 (+ 2145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:40.208169 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:40.208171 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:40.208550 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:40.208786 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:40.208787 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:40.208787 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:40.208920 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:40.209168 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:40.209173 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:40.209306 (+ 133us) client_negotiation.cc:769] Sending connection context
0925 15:20:40.209396 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 15:20:40.209530 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":87,"threads_started":1}
I20260925 15:20:40.210229 10478 heartbeater.cc:347] Connected to a master server at 127.3.211.254:38373
I20260925 15:20:40.210361 10478 heartbeater.cc:464] Registering TS with master...
I20260925 15:20:40.210567 10478 heartbeater.cc:511] Master 127.3.211.254:38373 requested a full tablet report, sending...
I20260925 15:20:40.211011 10018 ts_manager.cc:194] Registered new tserver with Master: fbc183e336c74b8a8d68f4624d9b375b (127.3.211.195:34379)
I20260925 15:20:40.211505 10018 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.211.195@KRBTEST.COM'} at 127.3.211.195:55983
I20260925 15:20:40.219126 3919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.211.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.211.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.211.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/oryx_127.3.211.253.keytab.
Entry for principal oryx/127.3.211.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/oryx_127.3.211.253.keytab.
I20260925 15:20:40.246942 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.253:42795
--webserver_interface=127.3.211.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/oryx_127.3.211.253.keytab
--principal=oryx/127.3.211.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:42075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.211.254:38373,127.3.211.253:42795
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:20:40.334815 10483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:40.335103 10483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:40.335141 10483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:40.336479 10483 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:20:40.336563 10483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:40.336587 10483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:40.336611 10483 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:20:40.336637 10483 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:20:40.338294 10483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:42075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.211.254:38373,127.3.211.253:42795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/oryx_127.3.211.253.keytab
--principal=oryx/127.3.211.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.253:42795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.211.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
W20260925 15:20:40.338481 10483 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.3.211.254:38373,127.3.211.253:42795'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 15:20:40.338653 10483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:40.338898 10483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:40.344863 10488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:40.344903 10489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:40.344862 10491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:40.345261 10483 server_base.cc:1034] running on GCE node
I20260925 15:20:40.345480 10483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:40.345737 10483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:40.346933 10483 hybrid_clock.cc:648] HybridClock initialized: now 1790349640346920 us; error 36 us; skew 500 ppm
Sep 25 15:20:40 dist-test-slave-4bk3 krb5kdc[9973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349640, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.211.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:40.349606 10483 init.cc:377] Logged in from keytab as oryx/127.3.211.253@KRBTEST.COM (short username oryx)
I20260925 15:20:40.350147 10483 webserver.cc:460] Webserver started at http://127.3.211.253:40161/ using document root <none> and password file <none>
I20260925 15:20:40.350404 10483 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:40.350476 10483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:40.350644 10483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:40.351837 10483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-1/data/instance:
uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3"
format_stamp: "Formatted at 2026-09-25 15:20:40 on dist-test-slave-4bk3"
server_key: "41037d1e4119e42b5d4b0ae5cdd847f8"
server_key_iv: "9f214a65fb7684abb7c530ad6b49270b"
server_key_version: "encryptionkey@0"
I20260925 15:20:40.352772 10483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-1/wal/instance:
uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3"
format_stamp: "Formatted at 2026-09-25 15:20:40 on dist-test-slave-4bk3"
server_key: "41037d1e4119e42b5d4b0ae5cdd847f8"
server_key_iv: "9f214a65fb7684abb7c530ad6b49270b"
server_key_version: "encryptionkey@0"
I20260925 15:20:40.355015 10483 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.001s
I20260925 15:20:40.356112 10498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:40.356408 10483 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:40.356525 10483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-1/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-1/wal
uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3"
format_stamp: "Formatted at 2026-09-25 15:20:40 on dist-test-slave-4bk3"
server_key: "41037d1e4119e42b5d4b0ae5cdd847f8"
server_key_iv: "9f214a65fb7684abb7c530ad6b49270b"
server_key_version: "encryptionkey@0"
I20260925 15:20:40.356621 10483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:40.382050 10483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:40.384653 10483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:40.384824 10483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:40.388870 10483 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.253:42795
I20260925 15:20:40.388906 10550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.253:42795 every 8 connection(s)
I20260925 15:20:40.389310 10483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-1/data/info.pb
I20260925 15:20:40.390406 10551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:40.392040 10551 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38373 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:20:40.392637 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 10483
Sep 25 15:20:40 dist-test-slave-4bk3 krb5kdc[9973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349640, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.211.253@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM
Sep 25 15:20:40 dist-test-slave-4bk3 krb5kdc[9973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.211.253@KRBTEST.COM
I20260925 15:20:40.402835 10211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:40.392864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47568 (local address 127.3.211.254:38373)
0925 15:20:40.392925 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:40.392928 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:40.393179 (+ 251us) server_negotiation.cc:407] Connection header received
0925 15:20:40.393898 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:40.393899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:40.393913 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:40.393981 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:40.394672 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.395199 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.396366 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.396677 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.396754 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:40.399381 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:40.399384 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:40.399384 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:40.399393 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:40.401078 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:40.401601 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:40.401602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:40.401602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:40.401681 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:40.401946 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:40.401947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:40.401947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:40.402168 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:40.402244 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:40.402598 (+ 354us) server_negotiation.cc:299] Negotiation successful
0925 15:20:40.402721 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 15:20:40.402997 10553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:40.392860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38373 (local address 127.0.0.1:47568)
0925 15:20:40.393045 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:40.393085 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:40.393753 (+ 668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:40.394013 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:40.394016 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:40.394342 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:40.394529 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:40.394533 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.395303 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.395304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:40.396256 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:40.396258 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.396770 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.396771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:40.396907 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:40.397621 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:40.397630 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:40.399279 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:40.401156 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:40.401161 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:40.401163 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:40.401507 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:40.401764 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:40.401765 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:40.401765 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:40.401882 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:40.402258 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:40.402287 (+ 29us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:40.402458 (+ 171us) client_negotiation.cc:769] Sending connection context
0925 15:20:40.402570 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 15:20:40.402743 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":4,"thread_start_us":104,"threads_started":1}
I20260925 15:20:40.402997 10555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:40.393232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52892 (local address 127.3.211.253:42795)
0925 15:20:40.393378 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:40.393383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:40.393448 (+ 65us) server_negotiation.cc:407] Connection header received
0925 15:20:40.393754 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:40.393764 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:40.393790 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:40.393898 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:40.394764 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.395844 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.396758 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.397180 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.397241 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:40.399878 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:40.399885 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:40.399885 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:40.399891 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:40.401662 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:40.402045 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:40.402047 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:40.402048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:40.402094 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:40.402304 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:40.402305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:40.402305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:40.402493 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:40.402649 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:40.402750 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 15:20:40.402814 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":74,"threads_started":1}
I20260925 15:20:40.403759 10551 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42795 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:20:40 dist-test-slave-4bk3 krb5kdc[9973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349640, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.211.253@KRBTEST.COM for oryx/127.3.211.253@KRBTEST.COM
I20260925 15:20:40.410907 10553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:40.404072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.253:42795 (local address 127.0.0.1:52906)
0925 15:20:40.404139 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:40.404150 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:40.404202 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:40.404361 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:40.404362 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:40.404507 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:40.404554 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:40.404556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.405215 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.405216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:40.405689 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:40.405690 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.406065 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.406066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:40.406127 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:40.406621 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:40.406629 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:40.408266 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:40.409906 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:40.409907 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:40.409908 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:40.410087 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:40.410288 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:40.410289 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:40.410289 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:40.410354 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:40.410627 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:40.410628 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:40.410670 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 15:20:40.410736 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 15:20:40.410827 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 15:20:40.411047 10555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:40.404117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52906 (local address 127.3.211.253:42795)
0925 15:20:40.404148 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:40.404151 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:40.404186 (+ 35us) server_negotiation.cc:407] Connection header received
0925 15:20:40.404266 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:40.404267 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:40.404278 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:40.404369 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:40.404634 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.405115 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.405785 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.405985 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.406069 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:40.408395 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:40.408396 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:40.408397 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:40.408406 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:40.409779 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:40.410169 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:40.410170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:40.410171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:40.410226 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:40.410419 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:40.410419 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:40.410420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:40.410536 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:40.410612 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:40.410808 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 15:20:40.410891 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 15:20:40.412709 10551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3: Bootstrap starting.
I20260925 15:20:40.413622 10551 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:40.413903 10551 log.cc:824] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3: Log is configured to *not* fsync() on all Append() calls
I20260925 15:20:40.414637 10551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3: No bootstrap required, opened a new log
I20260925 15:20:40.416060 10551 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38373 } } peers { permanent_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42795 } }
I20260925 15:20:40.416198 10551 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:40.416221 10551 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a04d4b30a3b47d0a897ed7f3afa45a3, State: Initialized, Role: FOLLOWER
I20260925 15:20:40.416404 10551 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38373 } } peers { permanent_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42795 } }
I20260925 15:20:40.416735 10551 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:20:40.416690 10557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38373 } } peers { permanent_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42795 } } }
I20260925 15:20:40.416807 10557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 15:20:40.418346 10568 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:20:40.418409 10568 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:20:40.420140 10483 master_runner.cc:428] Detected that this master 6a04d4b30a3b47d0a897ed7f3afa45a3 is joining an existing cluster
I20260925 15:20:40.420210 10483 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.211.253:42795
I20260925 15:20:40.428388 10555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:40.421537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52908 (local address 127.3.211.253:42795)
0925 15:20:40.421575 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:40.421576 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:40.421592 (+ 16us) server_negotiation.cc:407] Connection header received
0925 15:20:40.421658 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:40.421659 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:40.421670 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:40.421721 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:40.422298 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.422767 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.423725 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.424032 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.424083 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:40.425308 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:40.425309 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:40.425309 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:40.425316 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:40.427073 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:40.427497 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:40.427498 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:40.427499 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:40.427562 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:40.427789 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:40.427790 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:40.427790 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:40.427914 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:40.427964 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:40.428155 (+ 191us) server_negotiation.cc:299] Negotiation successful
0925 15:20:40.428238 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 15:20:40.428721 10575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:40.421373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.253:42795 (local address 127.0.0.1:52908)
0925 15:20:40.421542 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:40.421552 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:40.421593 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:40.421765 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:40.421766 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:40.422091 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:40.422216 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:40.422216 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.422830 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.422830 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:40.423551 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:40.423553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.424184 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.424185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:40.424253 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:40.424796 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:40.424802 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:40.425238 (+ 436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:40.427165 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:40.427165 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:40.427166 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:40.427430 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:40.427635 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:40.427636 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:40.427637 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:40.427713 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:40.428006 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:40.428007 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:40.428041 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 15:20:40.428519 (+ 478us) client_negotiation.cc:240] Negotiation successful
0925 15:20:40.428621 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":9,"thread_start_us":72,"threads_started":1}
I20260925 15:20:40.429428 10211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:40.421341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47570 (local address 127.3.211.254:38373)
0925 15:20:40.421392 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:40.421394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:40.421544 (+ 150us) server_negotiation.cc:407] Connection header received
0925 15:20:40.421735 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:40.421736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:40.421749 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:40.421806 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:40.422303 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.423087 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.425265 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.425573 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.425642 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:40.426672 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:40.426673 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:40.426674 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:40.426681 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:40.428123 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:40.428618 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:40.428619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:40.428619 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:40.428677 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:40.428886 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:40.428887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:40.428887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:40.429010 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:40.429052 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:40.429252 (+ 200us) server_negotiation.cc:299] Negotiation successful
0925 15:20:40.429338 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 15:20:40.429445 10574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:40.421304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38373 (local address 127.0.0.1:47570)
0925 15:20:40.421454 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:40.421464 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:40.421616 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:40.421828 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:40.421829 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:40.422082 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:40.422210 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:40.422212 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.423180 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.423182 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:40.425151 (+ 1969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:40.425155 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.425712 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.425714 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:40.425787 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:40.426166 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:40.426170 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:40.426575 (+ 405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:40.428224 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:40.428225 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:40.428225 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:40.428525 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:40.428744 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:40.428744 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:40.428745 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:40.428811 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:40.429095 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:40.429096 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:40.429139 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 15:20:40.429226 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 15:20:40.429338 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":5,"thread_start_us":98,"threads_started":1}
Sep 25 15:20:40 dist-test-slave-4bk3 krb5kdc[9973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349638, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.211.254@KRBTEST.COM for oryx/127.3.211.253@KRBTEST.COM
I20260925 15:20:40.441040 10555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:40.432399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52912 (local address 127.3.211.253:42795)
0925 15:20:40.432438 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:40.432440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:40.432658 (+ 218us) server_negotiation.cc:407] Connection header received
0925 15:20:40.432803 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:40.432804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:40.432815 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:40.432894 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:40.433306 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.434070 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.434917 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.435217 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.435290 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:40.437566 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:40.437568 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:40.437568 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:40.437577 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:40.439332 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:40.439886 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:40.439888 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:40.439888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:40.439961 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:40.440193 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:40.440194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:40.440194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:40.440374 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:40.440450 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:40.440829 (+ 379us) server_negotiation.cc:299] Negotiation successful
0925 15:20:40.440932 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 15:20:40.441150 10576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:40.432395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.253:42795 (local address 127.0.0.1:52912)
0925 15:20:40.432540 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:40.432584 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:40.432663 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:40.432898 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:40.432901 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:40.433174 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:40.433231 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:40.433233 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.434188 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.434190 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:40.434826 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:40.434828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.435329 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.435331 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:40.435399 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:40.435968 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:40.435974 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:40.437465 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:40.439452 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:40.439464 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:40.439467 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:40.439785 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:40.440051 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:40.440052 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:40.440052 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:40.440127 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:40.440526 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:40.440545 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:40.440645 (+ 100us) client_negotiation.cc:769] Sending connection context
0925 15:20:40.440788 (+ 143us) client_negotiation.cc:240] Negotiation successful
0925 15:20:40.441018 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":90,"threads_started":1}
I20260925 15:20:40.442044 10018 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.211.253:42795: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3"
member_type: NON_VOTER
last_known_addr {
host: "127.3.211.253"
port: 42795
}
attrs {
promote: true
}
}
dest_uuid: "b3860a4600f94b6fa48e2ac45ace80e8"
cas_config_opid_index: -1
I20260925 15:20:40.442443 10018 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38373 } } peers { permanent_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" member_type: NON_VOTER last_known_addr { host: "127.3.211.253" port: 42795 } attrs { promote: true } }
I20260925 15:20:40.443097 10578 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 6a04d4b30a3b47d0a897ed7f3afa45a3 (127.3.211.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38373 } } peers { permanent_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" member_type: NON_VOTER last_known_addr { host: "127.3.211.253" port: 42795 } attrs { promote: true } } }
I20260925 15:20:40.443457 10525 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:40.444214 10525 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3 [term 1 FOLLOWER]: Refusing update from remote peer b3860a4600f94b6fa48e2ac45ace80e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260925 15:20:40.444240 10578 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.211.253:42795
I20260925 15:20:40.444396 10578 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38373 } } peers { permanent_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" member_type: NON_VOTER last_known_addr { host: "127.3.211.253" port: 42795 } attrs { promote: true } } }
I20260925 15:20:40.444463 10578 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8 [sys.catalog]: This master's current role is: LEADER
I20260925 15:20:40.444726 10483 master.cc:537] Master@127.3.211.253:42795 shutting down...
I20260925 15:20:40.444871 10578 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" member_type: NON_VOTER last_known_addr { host: "127.3.211.253" port: 42795 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 15:20:40.445720 10003 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260925 15:20:40.446328 10003 consensus_peers.cc:487] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8 -> Peer 6a04d4b30a3b47d0a897ed7f3afa45a3 (127.3.211.253:42795): Couldn't send request to peer 6a04d4b30a3b47d0a897ed7f3afa45a3. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 15:20:40.447635 10483 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:20:40.447758 10483 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:20:40.447817 10483 tablet_replica.cc:330] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3: stopping tablet replica
I20260925 15:20:40.461305 10483 master.cc:552] Master@127.3.211.253:42795 shutdown complete.
I20260925 15:20:40.461393 10483 master_runner.cc:305] Clearing existing system tablet
I20260925 15:20:40.461529 10483 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:40.461589 10483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:40.462461 10483 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 15:20:40.463061 10583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:40.463214 10483 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:40.463274 10483 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 15:20:40.463326 10483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-1/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-1/wal
uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3"
format_stamp: "Formatted at 2026-09-25 15:20:40 on dist-test-slave-4bk3"
server_key: "41037d1e4119e42b5d4b0ae5cdd847f8"
server_key_iv: "9f214a65fb7684abb7c530ad6b49270b"
server_key_version: "encryptionkey@0"
I20260925 15:20:40.464082 10483 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:20:40.465502 10483 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 15:20:40.465569 10483 log.cc:1196] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 15:20:40.465794 10483 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3: Deleting consensus metadata
I20260925 15:20:40.465935 10483 master_runner.cc:315] Copying system tablet from 127.3.211.254:38373
I20260925 15:20:40.466614 10483 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3: tablet copy: Beginning tablet copy session from remote peer at address 127.3.211.254:38373
I20260925 15:20:40.474030 10211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:40.467237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47578 (local address 127.3.211.254:38373)
0925 15:20:40.467338 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:40.467340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:40.467422 (+ 82us) server_negotiation.cc:407] Connection header received
0925 15:20:40.467457 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:40.467458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:40.467471 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:40.467538 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:40.468038 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.468677 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.469842 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.470074 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.470160 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:40.471308 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:40.471309 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:40.471309 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:40.471316 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:40.472920 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:40.473326 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:40.473327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:40.473327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:40.473380 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:40.473572 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:40.473572 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:40.473573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:40.473682 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:40.473732 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:40.473892 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 15:20:40.473959 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 15:20:40.474097 10589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:40.467012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38373 (local address 127.0.0.1:47578)
0925 15:20:40.467294 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:40.467308 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:40.467364 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:40.467561 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:40.467562 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:40.467805 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:40.467884 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:40.467886 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.468767 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.468769 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:40.469729 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:40.469732 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.470163 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.470163 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:40.470242 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:40.470779 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:40.470785 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:40.471198 (+ 413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:40.473000 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:40.473002 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:40.473002 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:40.473244 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:40.473444 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:40.473444 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:40.473445 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:40.473511 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:40.473740 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:40.473742 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:40.473792 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 15:20:40.473866 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 15:20:40.473977 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":221,"threads_started":1}
I20260925 15:20:40.474403 10038 tablet_copy_service.cc:140] P b3860a4600f94b6fa48e2ac45ace80e8: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 6a04d4b30a3b47d0a897ed7f3afa45a3 ({username='oryx', principal='oryx/127.3.211.253@KRBTEST.COM'} at 127.0.0.1:47578)
I20260925 15:20:40.474495 10038 tablet_copy_service.cc:161] P b3860a4600f94b6fa48e2ac45ace80e8: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 6a04d4b30a3b47d0a897ed7f3afa45a3 at {username='oryx', principal='oryx/127.3.211.253@KRBTEST.COM'} at 127.0.0.1:47578: session id = 6a04d4b30a3b47d0a897ed7f3afa45a3-00000000000000000000000000000000
I20260925 15:20:40.475127 10038 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 15:20:40.475816 10483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 15:20:40.476871 10483 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3: tablet copy: Starting download of 0 data blocks...
I20260925 15:20:40.477030 10483 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3: tablet copy: Starting download of 1 WAL segments...
I20260925 15:20:40.477864 10483 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 15:20:40.478677 10038 tablet_copy_service.cc:342] P b3860a4600f94b6fa48e2ac45ace80e8: Request end of tablet copy session 6a04d4b30a3b47d0a897ed7f3afa45a3-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.211.253@KRBTEST.COM'} at 127.0.0.1:47578
I20260925 15:20:40.478751 10038 tablet_copy_service.cc:433] P b3860a4600f94b6fa48e2ac45ace80e8: ending tablet copy session 6a04d4b30a3b47d0a897ed7f3afa45a3-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 6a04d4b30a3b47d0a897ed7f3afa45a3
I20260925 15:20:40.481575 10483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 15:20:40.482827 10483 server_base.cc:1034] running on GCE node
W20260925 15:20:40.482837 10591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:40.482915 10590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:40.483033 10593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:40.483193 10483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:40.483364 10483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:40.484489 10483 hybrid_clock.cc:648] HybridClock initialized: now 1790349640484468 us; error 33 us; skew 500 ppm
Sep 25 15:20:40 dist-test-slave-4bk3 krb5kdc[9973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349640, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.211.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:40.485873 10483 init.cc:377] Logged in from keytab as oryx/127.3.211.253@KRBTEST.COM (short username oryx)
I20260925 15:20:40.486182 10483 webserver.cc:460] Webserver started at http://127.3.211.253:44927/ using document root <none> and password file <none>
I20260925 15:20:40.486325 10483 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:40.486361 10483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:40.487032 10483 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:20:40.487556 10600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:40.487695 10483 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:40.487743 10483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-1/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-1/wal
uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3"
format_stamp: "Formatted at 2026-09-25 15:20:40 on dist-test-slave-4bk3"
server_key: "41037d1e4119e42b5d4b0ae5cdd847f8"
server_key_iv: "9f214a65fb7684abb7c530ad6b49270b"
server_key_version: "encryptionkey@0"
I20260925 15:20:40.487803 10483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:40.493352 10483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:40.495132 10483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:40.498358 10483 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.253:42795
I20260925 15:20:40.498773 10652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.253:42795 every 8 connection(s)
I20260925 15:20:40.498823 10483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-1/data/info.pb
I20260925 15:20:40.499701 10653 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:20:40.500213 10653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3: Bootstrap starting.
I20260925 15:20:40.503269 10653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:20:40.503530 10653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3: Bootstrap complete.
I20260925 15:20:40.503806 10653 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38373 } } peers { permanent_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" member_type: NON_VOTER last_known_addr { host: "127.3.211.253" port: 42795 } attrs { promote: true } }
I20260925 15:20:40.503882 10653 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6a04d4b30a3b47d0a897ed7f3afa45a3, State: Initialized, Role: LEARNER
I20260925 15:20:40.503909 10653 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38373 } } peers { permanent_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" member_type: NON_VOTER last_known_addr { host: "127.3.211.253" port: 42795 } attrs { promote: true } }
I20260925 15:20:40.504102 10653 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:20:40.504133 10656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38373 } } peers { permanent_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" member_type: NON_VOTER last_known_addr { host: "127.3.211.253" port: 42795 } attrs { promote: true } } }
I20260925 15:20:40.504252 10656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3 [sys.catalog]: This master's current role is: LEARNER
I20260925 15:20:40.505705 10668 catalog_manager.cc:1259] Loaded cluster ID: f33e811646eb40de953a49f346770393
I20260925 15:20:40.505828 10668 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3: loading cluster ID for follower catalog manager: success
I20260925 15:20:40.507144 10668 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3: acquiring CA information for follower catalog manager: success
I20260925 15:20:40.507548 10668 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 15:20:40.516300 10662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:40.504889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52932 (local address 127.3.211.253:42795)
0925 15:20:40.508483 (+ 3594us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:40.508486 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:40.508501 (+ 15us) server_negotiation.cc:407] Connection header received
0925 15:20:40.508532 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:40.508533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:40.508546 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:40.508656 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:40.509255 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.510356 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.511228 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.511542 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.511740 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:40.513199 (+ 1459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:40.513200 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:40.513201 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:40.513209 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:40.514849 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:40.515362 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:40.515364 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:40.515364 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:40.515432 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:40.515688 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:40.515690 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:40.515691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:40.515838 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:40.515950 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:40.516094 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 15:20:40.516196 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3558,"thread_start_us":63,"threads_started":1}
WARNING: no policy specified for oryx/127.3.211.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.211.252@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.211.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/oryx_127.3.211.252.keytab.
Entry for principal oryx/127.3.211.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/oryx_127.3.211.252.keytab.
I20260925 15:20:40.543937 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.252:36373
--webserver_interface=127.3.211.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/oryx_127.3.211.252.keytab
--principal=oryx/127.3.211.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:42075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.211.254:38373,127.3.211.253:42795,127.3.211.252:36373
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:20:40.628547 10671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:40.628772 10671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:40.628798 10671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:40.630062 10671 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:20:40.630115 10671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:40.630131 10671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:40.630143 10671 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:20:40.630156 10671 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:20:40.631805 10671 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:42075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.211.254:38373,127.3.211.253:42795,127.3.211.252:36373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/oryx_127.3.211.252.keytab
--principal=oryx/127.3.211.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.252:36373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.211.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:40.632098 10671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:40.632381 10671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:40.637207 10679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:40.637207 10677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:40.637404 10676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:40.637764 10671 server_base.cc:1034] running on GCE node
I20260925 15:20:40.638068 10671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:40.638398 10671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:40.639568 10671 hybrid_clock.cc:648] HybridClock initialized: now 1790349640639544 us; error 37 us; skew 500 ppm
Sep 25 15:20:40 dist-test-slave-4bk3 krb5kdc[9973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349640, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.211.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:40.642297 10671 init.cc:377] Logged in from keytab as oryx/127.3.211.252@KRBTEST.COM (short username oryx)
I20260925 15:20:40.642817 10671 webserver.cc:460] Webserver started at http://127.3.211.252:40487/ using document root <none> and password file <none>
I20260925 15:20:40.643049 10671 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:40.643116 10671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:40.643276 10671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:40.644484 10671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-2/data/instance:
uuid: "881c1ed0940a4a7483c36869cba574f8"
format_stamp: "Formatted at 2026-09-25 15:20:40 on dist-test-slave-4bk3"
server_key: "0f6ae0292021f9cb92237d3b8c2afdae"
server_key_iv: "b4295c5dc8583af29727a465d14aee94"
server_key_version: "encryptionkey@0"
I20260925 15:20:40.644886 10671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-2/wal/instance:
uuid: "881c1ed0940a4a7483c36869cba574f8"
format_stamp: "Formatted at 2026-09-25 15:20:40 on dist-test-slave-4bk3"
server_key: "0f6ae0292021f9cb92237d3b8c2afdae"
server_key_iv: "b4295c5dc8583af29727a465d14aee94"
server_key_version: "encryptionkey@0"
I20260925 15:20:40.646341 10671 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 15:20:40.647079 10686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:40.647327 10671 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:20:40.647433 10671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-2/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-2/wal
uuid: "881c1ed0940a4a7483c36869cba574f8"
format_stamp: "Formatted at 2026-09-25 15:20:40 on dist-test-slave-4bk3"
server_key: "0f6ae0292021f9cb92237d3b8c2afdae"
server_key_iv: "b4295c5dc8583af29727a465d14aee94"
server_key_version: "encryptionkey@0"
I20260925 15:20:40.647513 10671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:40.678738 10671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:40.681193 10671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:40.681367 10671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:40.685494 10671 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.252:36373
I20260925 15:20:40.685565 10738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.252:36373 every 8 connection(s)
I20260925 15:20:40.685954 10671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-2/data/info.pb
I20260925 15:20:40.686918 10739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:40.688544 10739 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38373 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:20:40.689483 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 10671
Sep 25 15:20:40 dist-test-slave-4bk3 krb5kdc[9973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349640, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.211.252@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM
Sep 25 15:20:40 dist-test-slave-4bk3 krb5kdc[9973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.211.252@KRBTEST.COM
I20260925 15:20:40.698583 10211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:40.689509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47586 (local address 127.3.211.254:38373)
0925 15:20:40.689566 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:40.689568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:40.689728 (+ 160us) server_negotiation.cc:407] Connection header received
0925 15:20:40.690377 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:40.690379 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:40.690392 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:40.690463 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:40.691038 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.691530 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.692465 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.692656 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.692732 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:40.695430 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:40.695432 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:40.695432 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:40.695440 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:40.697169 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:40.697587 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:40.697588 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:40.697589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:40.697651 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:40.697891 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:40.697892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:40.697892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:40.698041 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:40.698131 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:40.698392 (+ 261us) server_negotiation.cc:299] Negotiation successful
0925 15:20:40.698489 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 15:20:40.698779 10741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:40.689343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38373 (local address 127.0.0.1:47586)
0925 15:20:40.689597 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:40.689643 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:40.690280 (+ 637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:40.690488 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:40.690491 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:40.690766 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:40.690921 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:40.690925 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.691628 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.691629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:40.692323 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:40.692369 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.692755 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.692756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:40.692870 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:40.693477 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:40.693487 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:40.695328 (+ 1841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:40.697250 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:40.697257 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:40.697259 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:40.697512 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:40.697725 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:40.697726 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:40.697726 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:40.697829 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:40.698124 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:40.698130 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:40.698262 (+ 132us) client_negotiation.cc:769] Sending connection context
0925 15:20:40.698373 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 15:20:40.698528 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":114,"threads_started":1}
I20260925 15:20:40.699294 10742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:40.689974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50362 (local address 127.3.211.252:36373)
0925 15:20:40.690121 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:40.690127 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:40.690164 (+ 37us) server_negotiation.cc:407] Connection header received
0925 15:20:40.690280 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:40.690286 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:40.690307 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:40.690429 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:40.691066 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.691962 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.692824 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.693215 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.693298 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:40.696246 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:40.696252 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:40.696253 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:40.696259 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:40.697901 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:40.698282 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:40.698285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:40.698285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:40.698348 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:40.698556 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:40.698557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:40.698557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:40.698770 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:40.698929 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:40.699102 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 15:20:40.699178 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":91,"threads_started":1}
I20260925 15:20:40.699456 10739 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42795 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:20:40 dist-test-slave-4bk3 krb5kdc[9973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349640, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.211.252@KRBTEST.COM for oryx/127.3.211.253@KRBTEST.COM
I20260925 15:20:40.706845 10662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:40.699801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52940 (local address 127.3.211.253:42795)
0925 15:20:40.699845 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:40.699847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:40.699857 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:20:40.699928 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:40.699929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:40.699942 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:40.700002 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:40.700215 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.700799 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.701503 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.701712 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.701787 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:40.703789 (+ 2002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:40.703790 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:40.703790 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:40.703797 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:40.705546 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:40.705978 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:40.705980 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:40.705980 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:40.706046 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:40.706314 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:40.706314 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:40.706315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:40.706445 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:40.706509 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:40.706701 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 15:20:40.706780 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 15:20:40.706846 10741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:40.699721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.253:42795 (local address 127.0.0.1:52940)
0925 15:20:40.699797 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:40.699808 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:40.699844 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:40.700011 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:40.700011 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:40.700122 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:40.700156 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:40.700158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.700889 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.700890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:40.701414 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:40.701416 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.701821 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.701822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:40.701878 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:40.702319 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:40.702323 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:40.703671 (+ 1348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:40.705645 (+ 1974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:40.705646 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:40.705647 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:40.705893 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:40.706144 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:40.706145 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:40.706146 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:40.706221 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:40.706538 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:40.706539 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:40.706611 (+ 72us) client_negotiation.cc:769] Sending connection context
0925 15:20:40.706682 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 15:20:40.706773 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 15:20:40.707450 10739 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.211.252" port: 36373 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:20:40 dist-test-slave-4bk3 krb5kdc[9973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349640, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.211.252@KRBTEST.COM for oryx/127.3.211.252@KRBTEST.COM
I20260925 15:20:40.716161 10741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:40.708016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.252:36373 (local address 127.0.0.1:50364)
0925 15:20:40.708077 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:40.708099 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:40.708150 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:40.708257 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:40.708258 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:40.708436 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:40.708473 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:40.708474 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.709528 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.709529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:40.710018 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:40.710019 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.710467 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.710468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:40.710516 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:40.710981 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:40.710986 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:40.712739 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:40.714864 (+ 2125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:40.714865 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:40.714866 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:40.715041 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:40.715296 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:40.715296 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:40.715297 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:40.715368 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:40.715871 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:40.715871 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:40.715914 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 15:20:40.715981 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 15:20:40.716075 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 15:20:40.716245 10742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:40.708050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50364 (local address 127.3.211.252:36373)
0925 15:20:40.708090 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:40.708092 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:40.708152 (+ 60us) server_negotiation.cc:407] Connection header received
0925 15:20:40.708188 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:40.708189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:40.708198 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:40.708247 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:40.708594 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.709412 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.710114 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.710398 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.710470 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:40.712855 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:40.712856 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:40.712856 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:40.712863 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:40.714674 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:40.715110 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:40.715111 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:40.715111 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:40.715198 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:40.715630 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:40.715631 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:40.715632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:40.715777 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:40.715843 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:40.716010 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 15:20:40.716141 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":20}
I20260925 15:20:40.718217 10739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8: Bootstrap starting.
I20260925 15:20:40.719199 10739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:40.719516 10739 log.cc:824] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8: Log is configured to *not* fsync() on all Append() calls
I20260925 15:20:40.720469 10739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8: No bootstrap required, opened a new log
I20260925 15:20:40.721966 10739 raft_consensus.cc:348] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38373 } } peers { permanent_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42795 } } peers { permanent_uuid: "881c1ed0940a4a7483c36869cba574f8" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 36373 } }
I20260925 15:20:40.722138 10739 raft_consensus.cc:374] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:40.722187 10739 raft_consensus.cc:729] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 881c1ed0940a4a7483c36869cba574f8, State: Initialized, Role: FOLLOWER
I20260925 15:20:40.722342 10739 consensus_queue.cc:260] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38373 } } peers { permanent_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42795 } } peers { permanent_uuid: "881c1ed0940a4a7483c36869cba574f8" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 36373 } }
I20260925 15:20:40.722664 10739 sys_catalog.cc:564] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:20:40.722638 10744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38373 } } peers { permanent_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42795 } } peers { permanent_uuid: "881c1ed0940a4a7483c36869cba574f8" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 36373 } } }
I20260925 15:20:40.722757 10744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 15:20:40.724453 10755 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:20:40.724572 10755 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:20:40.726827 10671 master_runner.cc:428] Detected that this master 881c1ed0940a4a7483c36869cba574f8 is joining an existing cluster
I20260925 15:20:40.726886 10671 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.211.252:36373
I20260925 15:20:40.736289 10742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:40.727974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50366 (local address 127.3.211.252:36373)
0925 15:20:40.728019 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:40.728021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:40.729018 (+ 997us) server_negotiation.cc:407] Connection header received
0925 15:20:40.729108 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:40.729109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:40.729117 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:40.729144 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:40.729527 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.730257 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.731369 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.731800 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.731975 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:40.733400 (+ 1425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:40.733402 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:40.733402 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:40.733409 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:40.735040 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:40.735475 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:40.735476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:40.735477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:40.735536 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:40.735729 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:40.735730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:40.735730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:40.735849 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:40.735924 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:40.736043 (+ 119us) server_negotiation.cc:299] Negotiation successful
0925 15:20:40.736160 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 15:20:40.736407 10763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:40.728067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.252:36373 (local address 127.0.0.1:50366)
0925 15:20:40.728948 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:40.728961 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:40.729043 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:40.729209 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:40.729210 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:40.729404 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:40.729452 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:40.729454 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.730367 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.730368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:40.731273 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:40.731275 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.732071 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.732072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:40.732181 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:40.732690 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:40.732697 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:40.733190 (+ 493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:40.735183 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:40.735184 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:40.735185 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:40.735382 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:40.735603 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:40.735604 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:40.735604 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:40.735661 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:40.735914 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:40.735915 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:40.735954 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 15:20:40.736020 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 15:20:40.736218 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":849,"mutex_wait_us":153,"thread_start_us":60,"threads_started":1}
I20260925 15:20:40.748354 10211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:40.728213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47590 (local address 127.3.211.254:38373)
0925 15:20:40.728256 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:40.728257 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:40.728396 (+ 139us) server_negotiation.cc:407] Connection header received
0925 15:20:40.728435 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:40.728435 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:40.728449 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:40.728510 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:40.728971 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.729770 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.730933 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.731253 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.731301 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:40.744937 (+ 13636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:40.744940 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:40.744940 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:40.744953 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:40.746927 (+ 1974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:40.747398 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:40.747399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:40.747400 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:40.747473 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:40.747719 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:40.747720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:40.747721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:40.747872 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:40.747953 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:40.748138 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 15:20:40.748226 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 15:20:40.748395 10762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:40.727952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38373 (local address 127.0.0.1:47590)
0925 15:20:40.728264 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:40.728275 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:40.728377 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:40.728525 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:40.728526 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:40.728746 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:40.728851 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:40.728855 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.729873 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.729874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:40.730816 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:40.730819 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.731401 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.731402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:40.731466 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:40.732019 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:40.732025 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:40.744759 (+ 12734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:40.747069 (+ 2310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:40.747070 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:40.747071 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:40.747308 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:40.747586 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:40.747587 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:40.747587 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:40.747653 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:40.747944 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:40.747945 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:40.748004 (+ 59us) client_negotiation.cc:769] Sending connection context
0925 15:20:40.748083 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 15:20:40.748211 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":113,"threads_started":1}
I20260925 15:20:40.748907 10761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:40.727995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.253:42795 (local address 127.0.0.1:52944)
0925 15:20:40.728424 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:40.728435 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:40.728619 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:40.729058 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:40.729060 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:40.729334 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:40.729403 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:40.729405 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.730367 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.730368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:40.731287 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:40.731289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.732071 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.732072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:40.732178 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:40.745014 (+ 12836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:40.745020 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:40.745426 (+ 406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:40.747742 (+ 2316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:40.747743 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:40.747743 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:40.747949 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:40.748148 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:40.748149 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:40.748149 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:40.748207 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:40.748550 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:40.748550 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:40.748594 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 15:20:40.748683 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 15:20:40.748791 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":32,"thread_start_us":82,"threads_started":1}
I20260925 15:20:40.748925 10662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:40.728214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52944 (local address 127.3.211.253:42795)
0925 15:20:40.728300 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:40.728302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:40.728489 (+ 187us) server_negotiation.cc:407] Connection header received
0925 15:20:40.728800 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:40.728802 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:40.728823 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:40.728998 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:40.729505 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.730266 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.731374 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.731816 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.731975 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:40.745595 (+ 13620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:40.745597 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:40.745598 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:40.745609 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:40.747676 (+ 2067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:40.748011 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:40.748012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:40.748012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:40.748075 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:40.748281 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:40.748282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:40.748282 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:40.748447 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:40.748499 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:40.748740 (+ 241us) server_negotiation.cc:299] Negotiation successful
0925 15:20:40.748833 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 15:20:40.750572 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 10080
Sep 25 15:20:40 dist-test-slave-4bk3 krb5kdc[9973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349638, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.211.254@KRBTEST.COM for oryx/127.3.211.252@KRBTEST.COM
I20260925 15:20:40.757481 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.193:32847
--local_ip_for_outbound_sockets=127.3.211.193
--tserver_master_addrs=127.3.211.254:38373
--webserver_port=41623
--webserver_interface=127.3.211.193
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/oryx_127.3.211.193.keytab
--principal=oryx/127.3.211.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:42075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.211.254:38373,127.3.211.253:42795
--tserver_master_addrs=127.3.211.254:38373,127.3.211.253:42795,127.3.211.252:36373 with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:20:40.761464 10576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:40.752304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.252:36373 (local address 127.0.0.1:50378)
0925 15:20:40.752405 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:40.752417 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:40.752468 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:40.752609 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:40.752610 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:40.752783 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:40.752828 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:40.752830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.753501 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.753502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:40.754162 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:40.754164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.754637 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.754638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:40.754711 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:40.755195 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:40.755201 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:40.757114 (+ 1913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:40.760172 (+ 3058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:40.760174 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:40.760175 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:40.760530 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:40.760764 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:40.760765 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:40.760766 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:40.760833 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:40.761106 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:40.761107 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:40.761156 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 15:20:40.761237 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 15:20:40.761362 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260925 15:20:40.761464 10742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:40.752360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50378 (local address 127.3.211.252:36373)
0925 15:20:40.752405 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:40.752407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:40.752452 (+ 45us) server_negotiation.cc:407] Connection header received
0925 15:20:40.752542 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:40.752542 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:40.752552 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:40.752596 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:40.752904 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.753420 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.754248 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.754544 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.754623 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:40.757238 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:40.757240 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:40.757241 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:40.757248 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:40.760021 (+ 2773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:40.760621 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:40.760623 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:40.760623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:40.760688 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:40.760908 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:40.760909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:40.760910 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:40.761034 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:40.761101 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:40.761275 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 15:20:40.761361 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 15:20:40.762389 10018 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.211.252:36373: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "881c1ed0940a4a7483c36869cba574f8"
member_type: NON_VOTER
last_known_addr {
host: "127.3.211.252"
port: 36373
}
attrs {
promote: true
}
}
dest_uuid: "b3860a4600f94b6fa48e2ac45ace80e8"
cas_config_opid_index: 5
I20260925 15:20:40.762665 10018 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38373 } } peers { permanent_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" member_type: NON_VOTER last_known_addr { host: "127.3.211.253" port: 42795 } attrs { promote: true } } peers { permanent_uuid: "881c1ed0940a4a7483c36869cba574f8" member_type: NON_VOTER last_known_addr { host: "127.3.211.252" port: 36373 } attrs { promote: true } }
I20260925 15:20:40.763021 10579 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 881c1ed0940a4a7483c36869cba574f8 (127.3.211.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38373 } } peers { permanent_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" member_type: NON_VOTER last_known_addr { host: "127.3.211.253" port: 42795 } attrs { promote: true } } peers { permanent_uuid: "881c1ed0940a4a7483c36869cba574f8" member_type: NON_VOTER last_known_addr { host: "127.3.211.252" port: 36373 } attrs { promote: true } } }
I20260925 15:20:40.763352 10713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:40.763888 10579 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.211.252:36373
I20260925 15:20:40.763972 10579 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38373 } } peers { permanent_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" member_type: NON_VOTER last_known_addr { host: "127.3.211.253" port: 42795 } attrs { promote: true } } peers { permanent_uuid: "881c1ed0940a4a7483c36869cba574f8" member_type: NON_VOTER last_known_addr { host: "127.3.211.252" port: 36373 } attrs { promote: true } } }
I20260925 15:20:40.764048 10579 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8 [sys.catalog]: This master's current role is: LEADER
I20260925 15:20:40.764146 10671 master.cc:537] Master@127.3.211.252:36373 shutting down...
I20260925 15:20:40.764524 10713 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [term 1 FOLLOWER]: Refusing update from remote peer b3860a4600f94b6fa48e2ac45ace80e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 15:20:40.764885 10578 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "881c1ed0940a4a7483c36869cba574f8" member_type: NON_VOTER last_known_addr { host: "127.3.211.252" port: 36373 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 15:20:40.765590 10002 consensus_peers.cc:487] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8 -> Peer 881c1ed0940a4a7483c36869cba574f8 (127.3.211.252:36373): Couldn't send request to peer 881c1ed0940a4a7483c36869cba574f8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 15:20:40.767871 10671 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:20:40.767985 10671 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:20:40.768035 10671 tablet_replica.cc:330] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8: stopping tablet replica
I20260925 15:20:40.770939 10662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:40.764840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52950 (local address 127.3.211.253:42795)
0925 15:20:40.764887 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:40.764890 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:40.766478 (+ 1588us) server_negotiation.cc:407] Connection header received
0925 15:20:40.766582 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:40.766583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:40.766609 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:40.766651 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:20:40.766893 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.767605 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.769732 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.770131 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.770173 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:40.770223 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:40.770791 (+ 568us) server_negotiation.cc:299] Negotiation successful
0925 15:20:40.770846 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 15:20:40.771529 10576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:40.764734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.253:42795 (local address 127.0.0.1:52950)
0925 15:20:40.766412 (+ 1678us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:40.766426 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:40.766507 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:40.766754 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:40.766755 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:40.766757 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:20:40.766812 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:40.766814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.767805 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.767806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:40.769027 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:40.769031 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.770385 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.770386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:40.770463 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:40.770492 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 15:20:40.771267 (+ 775us) client_negotiation.cc:240] Negotiation successful
0925 15:20:40.771408 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1247}
I20260925 15:20:40.772028 10627 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3 [term 1 LEARNER]: Refusing update from remote peer b3860a4600f94b6fa48e2ac45ace80e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 15:20:40.772281 10671 master.cc:552] Master@127.3.211.252:36373 shutdown complete.
I20260925 15:20:40.772260 10578 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" member_type: NON_VOTER last_known_addr { host: "127.3.211.253" port: 42795 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 15:20:40.772316 10671 master_runner.cc:305] Clearing existing system tablet
I20260925 15:20:40.772492 10671 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:40.772538 10671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:40.772933 10627 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 881c1ed0940a4a7483c36869cba574f8 (127.3.211.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38373 } } peers { permanent_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" member_type: NON_VOTER last_known_addr { host: "127.3.211.253" port: 42795 } attrs { promote: true } } peers { permanent_uuid: "881c1ed0940a4a7483c36869cba574f8" member_type: NON_VOTER last_known_addr { host: "127.3.211.252" port: 36373 } attrs { promote: true } } }
I20260925 15:20:40.773286 10671 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260925 15:20:40.773717 10767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:40.773805 10671 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:40.773844 10671 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:40.773864 10671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-2/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-2/wal
uuid: "881c1ed0940a4a7483c36869cba574f8"
format_stamp: "Formatted at 2026-09-25 15:20:40 on dist-test-slave-4bk3"
server_key: "0f6ae0292021f9cb92237d3b8c2afdae"
server_key_iv: "b4295c5dc8583af29727a465d14aee94"
server_key_version: "encryptionkey@0"
I20260925 15:20:40.774417 10656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3860a4600f94b6fa48e2ac45ace80e8. Latest consensus state: current_term: 1 leader_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38373 } } peers { permanent_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" member_type: NON_VOTER last_known_addr { host: "127.3.211.253" port: 42795 } attrs { promote: true } } peers { permanent_uuid: "881c1ed0940a4a7483c36869cba574f8" member_type: NON_VOTER last_known_addr { host: "127.3.211.252" port: 36373 } attrs { promote: true } } }
I20260925 15:20:40.774501 10656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3 [sys.catalog]: This master's current role is: LEARNER
I20260925 15:20:40.774539 10656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38373 } } peers { permanent_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" member_type: NON_VOTER last_known_addr { host: "127.3.211.253" port: 42795 } attrs { promote: true } } peers { permanent_uuid: "881c1ed0940a4a7483c36869cba574f8" member_type: NON_VOTER last_known_addr { host: "127.3.211.252" port: 36373 } attrs { promote: true } } }
I20260925 15:20:40.774573 10656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3 [sys.catalog]: This master's current role is: LEARNER
I20260925 15:20:40.774721 10671 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:20:40.775943 10671 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 15:20:40.776024 10671 log.cc:1196] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 15:20:40.776233 10671 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8: Deleting consensus metadata
I20260925 15:20:40.776420 10671 master_runner.cc:315] Copying system tablet from 127.3.211.254:38373
I20260925 15:20:40.777083 10671 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8: tablet copy: Beginning tablet copy session from remote peer at address 127.3.211.254:38373
I20260925 15:20:40.783290 10211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:40.777466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47598 (local address 127.3.211.254:38373)
0925 15:20:40.777525 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:40.777528 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:40.777608 (+ 80us) server_negotiation.cc:407] Connection header received
0925 15:20:40.777681 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:40.777682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:40.777693 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:40.777738 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:40.778061 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.778570 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.779294 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.779501 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.779588 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:40.780729 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:40.780730 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:40.780731 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:40.780738 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:40.782121 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:40.782503 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:40.782504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:40.782504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:40.782562 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:40.782755 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:40.782756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:40.782757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:40.782872 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:40.782946 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:40.783094 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 15:20:40.783183 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 15:20:40.783316 10774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:40.777420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38373 (local address 127.0.0.1:47598)
0925 15:20:40.777531 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:40.777542 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:40.777608 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:40.777752 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:40.777753 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:40.777929 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:40.777985 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:40.777986 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.778671 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.778672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:40.779213 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:40.779214 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.779588 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.779590 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:40.779657 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:40.780142 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:40.780148 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:40.780620 (+ 472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:40.782233 (+ 1613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:40.782235 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:40.782235 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:40.782417 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:40.782628 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:40.782629 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:40.782629 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:40.782687 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:40.782936 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:40.782937 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:40.782985 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 15:20:40.783069 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 15:20:40.783186 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":70,"threads_started":1}
I20260925 15:20:40.783684 10038 tablet_copy_service.cc:140] P b3860a4600f94b6fa48e2ac45ace80e8: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 881c1ed0940a4a7483c36869cba574f8 ({username='oryx', principal='oryx/127.3.211.252@KRBTEST.COM'} at 127.0.0.1:47598)
I20260925 15:20:40.783735 10038 tablet_copy_service.cc:161] P b3860a4600f94b6fa48e2ac45ace80e8: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 881c1ed0940a4a7483c36869cba574f8 at {username='oryx', principal='oryx/127.3.211.252@KRBTEST.COM'} at 127.0.0.1:47598: session id = 881c1ed0940a4a7483c36869cba574f8-00000000000000000000000000000000
I20260925 15:20:40.784166 10038 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 15:20:40.784801 10671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 15:20:40.784976 10578 raft_consensus.cc:1050] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8: attempting to promote NON_VOTER 6a04d4b30a3b47d0a897ed7f3afa45a3 to VOTER
I20260925 15:20:40.785195 10578 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38373 } } peers { permanent_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42795 } attrs { promote: false } } peers { permanent_uuid: "881c1ed0940a4a7483c36869cba574f8" member_type: NON_VOTER last_known_addr { host: "127.3.211.252" port: 36373 } attrs { promote: true } }
I20260925 15:20:40.785892 10576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:40.785762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.252:36373 (local address 127.0.0.1:50392)
0925 15:20:40.785813 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:40.785860 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.252:36373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 15:20:40.786152 10627 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3 [term 1 LEARNER]: Refusing update from remote peer b3860a4600f94b6fa48e2ac45ace80e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 15:20:40.786351 10671 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8: tablet copy: Starting download of 0 data blocks...
I20260925 15:20:40.786369 10579 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42795 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 15:20:40.786505 10671 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8: tablet copy: Starting download of 1 WAL segments...
I20260925 15:20:40.786764 10576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:40.786681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.252:36373 (local address 127.0.0.1:50400)
0925 15:20:40.786715 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:40.786738 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.252:36373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260925 15:20:40.786825 10002 consensus_peers.cc:487] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8 -> Peer 881c1ed0940a4a7483c36869cba574f8 (127.3.211.252:36373): Couldn't send request to peer 881c1ed0940a4a7483c36869cba574f8. Status: Network error: Client connection negotiation failed: client connection to 127.3.211.252:36373: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 15:20:40.787389 10579 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 6a04d4b30a3b47d0a897ed7f3afa45a3 (127.3.211.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38373 } } peers { permanent_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42795 } attrs { promote: false } } peers { permanent_uuid: "881c1ed0940a4a7483c36869cba574f8" member_type: NON_VOTER last_known_addr { host: "127.3.211.252" port: 36373 } attrs { promote: true } } }
I20260925 15:20:40.788641 10578 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38373 } } peers { permanent_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42795 } attrs { promote: false } } peers { permanent_uuid: "881c1ed0940a4a7483c36869cba574f8" member_type: NON_VOTER last_known_addr { host: "127.3.211.252" port: 36373 } attrs { promote: true } } }
I20260925 15:20:40.788722 10578 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8 [sys.catalog]: This master's current role is: LEADER
I20260925 15:20:40.788699 10627 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 6a04d4b30a3b47d0a897ed7f3afa45a3 (127.3.211.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38373 } } peers { permanent_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42795 } attrs { promote: false } } peers { permanent_uuid: "881c1ed0940a4a7483c36869cba574f8" member_type: NON_VOTER last_known_addr { host: "127.3.211.252" port: 36373 } attrs { promote: true } } }
I20260925 15:20:40.788303 10579 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38373 } } peers { permanent_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42795 } attrs { promote: false } } peers { permanent_uuid: "881c1ed0940a4a7483c36869cba574f8" member_type: NON_VOTER last_known_addr { host: "127.3.211.252" port: 36373 } attrs { promote: true } } }
I20260925 15:20:40.788898 10579 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8 [sys.catalog]: This master's current role is: LEADER
I20260925 15:20:40.788980 10671 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 15:20:40.789731 10768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38373 } } peers { permanent_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42795 } attrs { promote: false } } peers { permanent_uuid: "881c1ed0940a4a7483c36869cba574f8" member_type: NON_VOTER last_known_addr { host: "127.3.211.252" port: 36373 } attrs { promote: true } } }
I20260925 15:20:40.789829 10768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:20:40.789935 10038 tablet_copy_service.cc:342] P b3860a4600f94b6fa48e2ac45ace80e8: Request end of tablet copy session 881c1ed0940a4a7483c36869cba574f8-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.211.252@KRBTEST.COM'} at 127.0.0.1:47598
I20260925 15:20:40.789978 10038 tablet_copy_service.cc:433] P b3860a4600f94b6fa48e2ac45ace80e8: ending tablet copy session 881c1ed0940a4a7483c36869cba574f8-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 881c1ed0940a4a7483c36869cba574f8
I20260925 15:20:40.792640 10671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:40.793846 10776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:40.793883 10777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:40.794014 10779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:40.796571 10671 server_base.cc:1034] running on GCE node
I20260925 15:20:40.796697 10671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:40.796878 10671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:40.798022 10671 hybrid_clock.cc:648] HybridClock initialized: now 1790349640798006 us; error 31 us; skew 500 ppm
Sep 25 15:20:40 dist-test-slave-4bk3 krb5kdc[9973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349640, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.211.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:40.799530 10671 init.cc:377] Logged in from keytab as oryx/127.3.211.252@KRBTEST.COM (short username oryx)
I20260925 15:20:40.799868 10671 webserver.cc:460] Webserver started at http://127.3.211.252:35489/ using document root <none> and password file <none>
I20260925 15:20:40.800006 10671 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:40.800047 10671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:40.800864 10671 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:40.801486 10786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:40.801682 10671 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:20:40.801738 10671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-2/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-2/wal
uuid: "881c1ed0940a4a7483c36869cba574f8"
format_stamp: "Formatted at 2026-09-25 15:20:40 on dist-test-slave-4bk3"
server_key: "0f6ae0292021f9cb92237d3b8c2afdae"
server_key_iv: "b4295c5dc8583af29727a465d14aee94"
server_key_version: "encryptionkey@0"
I20260925 15:20:40.801810 10671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:40.824822 10671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:40.826696 10671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:40.831339 10671 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.252:36373
I20260925 15:20:40.831795 10671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/master-2/data/info.pb
I20260925 15:20:40.831357 10838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.252:36373 every 8 connection(s)
I20260925 15:20:40.832775 10839 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:20:40.833173 10839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8: Bootstrap starting.
I20260925 15:20:40.835669 10839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:20:40.835932 10839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8: Bootstrap complete.
I20260925 15:20:40.836145 10839 raft_consensus.cc:348] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38373 } } peers { permanent_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" member_type: NON_VOTER last_known_addr { host: "127.3.211.253" port: 42795 } attrs { promote: true } } peers { permanent_uuid: "881c1ed0940a4a7483c36869cba574f8" member_type: NON_VOTER last_known_addr { host: "127.3.211.252" port: 36373 } attrs { promote: true } }
I20260925 15:20:40.836236 10839 raft_consensus.cc:729] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 881c1ed0940a4a7483c36869cba574f8, State: Initialized, Role: LEARNER
I20260925 15:20:40.836302 10839 consensus_queue.cc:260] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38373 } } peers { permanent_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" member_type: NON_VOTER last_known_addr { host: "127.3.211.253" port: 42795 } attrs { promote: true } } peers { permanent_uuid: "881c1ed0940a4a7483c36869cba574f8" member_type: NON_VOTER last_known_addr { host: "127.3.211.252" port: 36373 } attrs { promote: true } }
I20260925 15:20:40.836531 10839 sys_catalog.cc:564] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:20:40.836524 10842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38373 } } peers { permanent_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" member_type: NON_VOTER last_known_addr { host: "127.3.211.253" port: 42795 } attrs { promote: true } } peers { permanent_uuid: "881c1ed0940a4a7483c36869cba574f8" member_type: NON_VOTER last_known_addr { host: "127.3.211.252" port: 36373 } attrs { promote: true } } }
I20260925 15:20:40.836727 10842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [sys.catalog]: This master's current role is: LEARNER
I20260925 15:20:40.838110 10853 catalog_manager.cc:1259] Loaded cluster ID: f33e811646eb40de953a49f346770393
I20260925 15:20:40.838207 10853 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8: loading cluster ID for follower catalog manager: success
I20260925 15:20:40.839361 10853 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8: acquiring CA information for follower catalog manager: success
I20260925 15:20:40.839663 10853 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 15:20:40.888712 10764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:40.889053 10764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:40.889104 10764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:40.891111 10764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:40.891199 10764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:40.891245 10764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.193
I20260925 15:20:40.893652 10764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:42075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/oryx_127.3.211.193.keytab
--principal=oryx/127.3.211.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.193:32847
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.211.193
--webserver_port=41623
--tserver_master_addrs=127.3.211.254:38373,127.3.211.253:42795,127.3.211.252:36373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.193
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:40.893956 10764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:40.894248 10764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:40.899834 10861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:40.900053 10764 server_base.cc:1034] running on GCE node
W20260925 15:20:40.899850 10858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:40.899849 10859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:40.900435 10764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:40.900744 10764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:40.901944 10764 hybrid_clock.cc:648] HybridClock initialized: now 1790349640901930 us; error 33 us; skew 500 ppm
Sep 25 15:20:40 dist-test-slave-4bk3 krb5kdc[9973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349640, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.211.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:40.905494 10764 init.cc:377] Logged in from keytab as oryx/127.3.211.193@KRBTEST.COM (short username oryx)
I20260925 15:20:40.905961 10764 webserver.cc:460] Webserver started at http://127.3.211.193:41623/ using document root <none> and password file <none>
I20260925 15:20:40.906237 10764 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:40.906338 10764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:40.908073 10764 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:20:40.908865 10868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:40.909055 10764 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 15:20:40.909159 10764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-0/wal
uuid: "bf6c0d3ea9e342d6960a6f8a5d1e7e26"
format_stamp: "Formatted at 2026-09-25 15:20:39 on dist-test-slave-4bk3"
server_key: "3f20fad442104bdcc87156ef5e45468f"
server_key_iv: "cea7526e4382ddf1b0879d046696c9f5"
server_key_version: "encryptionkey@0"
I20260925 15:20:40.909539 10764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:40.919140 10764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:40.921594 10764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:40.921774 10764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:40.922017 10764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:20:40.922334 10764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:20:40.922387 10764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:40.922432 10764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:20:40.922457 10764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:40.928843 10764 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.193:32847
I20260925 15:20:40.928951 10981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.193:32847 every 8 connection(s)
I20260925 15:20:40.929359 10764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:20:40.935130 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 10764
I20260925 15:20:40.935308 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 10215
Sep 25 15:20:40 dist-test-slave-4bk3 krb5kdc[9973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349640, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.211.193@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM
Sep 25 15:20:40 dist-test-slave-4bk3 krb5kdc[9973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349640, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.211.193@KRBTEST.COM for oryx/127.3.211.252@KRBTEST.COM
Sep 25 15:20:40 dist-test-slave-4bk3 krb5kdc[9973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349640, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.211.193@KRBTEST.COM for oryx/127.3.211.253@KRBTEST.COM
I20260925 15:20:40.941496 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.194:44803
--local_ip_for_outbound_sockets=127.3.211.194
--tserver_master_addrs=127.3.211.254:38373
--webserver_port=46841
--webserver_interface=127.3.211.194
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/oryx_127.3.211.194.keytab
--principal=oryx/127.3.211.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:42075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.211.254:38373,127.3.211.253:42795
--tserver_master_addrs=127.3.211.254:38373,127.3.211.253:42795,127.3.211.252:36373 with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:20:40.941622 10211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:40.930620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:36899 (local address 127.3.211.254:38373)
0925 15:20:40.930668 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:40.930670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:40.930689 (+ 19us) server_negotiation.cc:407] Connection header received
0925 15:20:40.931223 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:40.931225 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:40.931237 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:40.931336 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:40.932171 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.932937 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.933945 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.934154 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.934250 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:40.937755 (+ 3505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:40.937758 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:40.937758 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:40.937767 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:40.939855 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:40.940433 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:40.940434 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:40.940435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:40.940496 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:40.940797 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:40.940798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:40.940798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:40.940949 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:40.940992 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:40.941436 (+ 444us) server_negotiation.cc:299] Negotiation successful
0925 15:20:40.941525 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 15:20:40.941847 10989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:40.930228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38373 (local address 127.3.211.193:36899)
0925 15:20:40.930570 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:40.930580 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:40.931089 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:40.931433 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:40.931439 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:40.932003 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:40.932062 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:40.932064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.933033 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.933034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:40.933859 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:40.933861 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.934260 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.934261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:40.934397 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:40.935090 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:40.935101 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:40.937632 (+ 2531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:40.939948 (+ 2316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:40.939957 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:40.939959 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:40.940298 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:40.940567 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:40.940568 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:40.940568 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:40.940724 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:40.941038 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:40.941043 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:40.941226 (+ 183us) client_negotiation.cc:769] Sending connection context
0925 15:20:40.941413 (+ 187us) client_negotiation.cc:240] Negotiation successful
0925 15:20:40.941591 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":5,"spinlock_wait_cycles":8576,"thread_start_us":142,"threads_started":1}
I20260925 15:20:40.942286 10983 heartbeater.cc:347] Connected to a master server at 127.3.211.254:38373
I20260925 15:20:40.942310 10662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:40.930558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:56917 (local address 127.3.211.253:42795)
0925 15:20:40.930611 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:40.930614 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:40.931360 (+ 746us) server_negotiation.cc:407] Connection header received
0925 15:20:40.931432 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:40.931433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:40.931446 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:40.931493 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:40.932143 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.932784 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.933961 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.934273 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.934312 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:40.939047 (+ 4735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:40.939049 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:40.939049 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:40.939058 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:40.941045 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:40.941438 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:40.941439 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:40.941440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:40.941492 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:40.941695 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:40.941696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:40.941696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:40.941859 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:40.941903 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:40.942101 (+ 198us) server_negotiation.cc:299] Negotiation successful
0925 15:20:40.942183 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 15:20:40.942392 10983 heartbeater.cc:464] Registering TS with master...
I20260925 15:20:40.942570 10983 heartbeater.cc:511] Master 127.3.211.254:38373 requested a full tablet report, sending...
I20260925 15:20:40.942713 10988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:40.930238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.253:42795 (local address 127.3.211.193:56917)
0925 15:20:40.931296 (+ 1058us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:40.931304 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:40.931367 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:40.931510 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:40.931511 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:40.931825 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:40.932025 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:40.932030 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.932888 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.932889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:40.933860 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:40.933862 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.934389 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.934390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:40.934458 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:40.935161 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:40.935166 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:40.938969 (+ 3803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:40.941134 (+ 2165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:40.941135 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:40.941135 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:40.941352 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:40.941567 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:40.941568 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:40.941568 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:40.941630 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:40.941950 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:40.941951 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:40.941994 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 15:20:40.942396 (+ 402us) client_negotiation.cc:240] Negotiation successful
0925 15:20:40.942522 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1026,"spinlock_wait_cycles":41344,"thread_start_us":127,"threads_started":1}
I20260925 15:20:40.942996 10018 ts_manager.cc:194] Re-registered known tserver with Master: bf6c0d3ea9e342d6960a6f8a5d1e7e26 (127.3.211.193:32847)
I20260925 15:20:40.943536 10018 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.211.193@KRBTEST.COM'} at 127.3.211.193:36899
I20260925 15:20:40.943678 10984 heartbeater.cc:347] Connected to a master server at 127.3.211.253:42795
I20260925 15:20:40.943746 10984 heartbeater.cc:464] Registering TS with master...
I20260925 15:20:40.943825 10984 heartbeater.cc:511] Master 127.3.211.253:42795 requested a full tablet report, sending...
I20260925 15:20:40.944219 10617 ts_manager.cc:194] Registered new tserver with Master: bf6c0d3ea9e342d6960a6f8a5d1e7e26 (127.3.211.193:32847)
I20260925 15:20:40.944748 10986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:40.930206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:60385 (local address 127.3.211.252:36373)
0925 15:20:40.930344 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:40.930347 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:40.930564 (+ 217us) server_negotiation.cc:407] Connection header received
0925 15:20:40.931404 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:40.931406 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:40.931418 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:40.931499 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:40.932479 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.933498 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.934481 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.934755 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.934843 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:40.938459 (+ 3616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:40.938461 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:40.938461 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:40.938470 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:40.940633 (+ 2163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:40.941153 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:40.941154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:40.941155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:40.943779 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:40.944052 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:40.944053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:40.944053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:40.944214 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:40.944292 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:40.944541 (+ 249us) server_negotiation.cc:299] Negotiation successful
0925 15:20:40.944643 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":56,"threads_started":1}
I20260925 15:20:40.944978 10987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:40.930228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.252:36373 (local address 127.3.211.193:60385)
0925 15:20:40.930420 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:40.930469 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:40.931091 (+ 622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:40.932171 (+ 1080us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:40.932171 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:40.932319 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:40.932399 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:40.932401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.933588 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.933589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:40.934404 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:40.934405 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:40.934845 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:40.934846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:40.934907 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:40.935289 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:40.935293 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:40.938339 (+ 3046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:40.940726 (+ 2387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:40.940727 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:40.940727 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:40.941069 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:40.943889 (+ 2820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:40.943889 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:40.943890 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:40.943973 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:40.944289 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:40.944290 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:40.944410 (+ 120us) client_negotiation.cc:769] Sending connection context
0925 15:20:40.944500 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 15:20:40.944633 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":48,"thread_start_us":93,"threads_started":1}
I20260925 15:20:40.945420 10982 heartbeater.cc:347] Connected to a master server at 127.3.211.252:36373
I20260925 15:20:40.945482 10982 heartbeater.cc:464] Registering TS with master...
I20260925 15:20:40.945559 10982 heartbeater.cc:511] Master 127.3.211.252:36373 requested a full tablet report, sending...
I20260925 15:20:40.945902 10803 ts_manager.cc:194] Registered new tserver with Master: bf6c0d3ea9e342d6960a6f8a5d1e7e26 (127.3.211.193:32847)
W20260925 15:20:41.029628 10990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:41.029861 10990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:41.029888 10990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:41.031080 10990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:41.031126 10990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:41.031155 10990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.194
I20260925 15:20:41.032687 10990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:42075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/oryx_127.3.211.194.keytab
--principal=oryx/127.3.211.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.194:44803
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.211.194
--webserver_port=46841
--tserver_master_addrs=127.3.211.254:38373,127.3.211.253:42795,127.3.211.252:36373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.194
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:41.032896 10990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:41.033092 10990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:41.038407 10995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:41.038407 10998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:41.038422 10996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:41.038591 10990 server_base.cc:1034] running on GCE node
I20260925 15:20:41.038894 10990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:41.039157 10990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:41.040375 10990 hybrid_clock.cc:648] HybridClock initialized: now 1790349641040222 us; error 134 us; skew 500 ppm
Sep 25 15:20:41 dist-test-slave-4bk3 krb5kdc[9973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349641, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.211.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:41.043466 10990 init.cc:377] Logged in from keytab as oryx/127.3.211.194@KRBTEST.COM (short username oryx)
I20260925 15:20:41.043943 10990 webserver.cc:460] Webserver started at http://127.3.211.194:46841/ using document root <none> and password file <none>
I20260925 15:20:41.044178 10990 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:41.044256 10990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:41.045778 10990 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:41.046363 11005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:41.046542 10990 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:20:41.046636 10990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-1/wal
uuid: "7921060cf7c04ccf844a869c26b58c4a"
format_stamp: "Formatted at 2026-09-25 15:20:39 on dist-test-slave-4bk3"
server_key: "c2dd285cb4fcdd087bc663304a7cb4ca"
server_key_iv: "dddf2e047dcf47c9e97a37ee60135893"
server_key_version: "encryptionkey@0"
I20260925 15:20:41.047008 10990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:41.081820 10990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:41.084208 10990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:41.084415 10990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:41.084699 10990 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:20:41.085018 10990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:20:41.085069 10990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:41.085110 10990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:20:41.085140 10990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:41.091943 10990 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.194:44803
I20260925 15:20:41.092006 11118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.194:44803 every 8 connection(s)
I20260925 15:20:41.092456 10990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
I20260925 15:20:41.098595 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 10990
I20260925 15:20:41.098790 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 10349
Sep 25 15:20:41 dist-test-slave-4bk3 krb5kdc[9973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349641, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.211.194@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM
Sep 25 15:20:41 dist-test-slave-4bk3 krb5kdc[9973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349641, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.211.194@KRBTEST.COM for oryx/127.3.211.252@KRBTEST.COM
Sep 25 15:20:41 dist-test-slave-4bk3 krb5kdc[9973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349641, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.211.194@KRBTEST.COM for oryx/127.3.211.253@KRBTEST.COM
I20260925 15:20:41.104995 10662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:41.093590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:43431 (local address 127.3.211.253:42795)
0925 15:20:41.093640 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:41.093642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:41.093829 (+ 187us) server_negotiation.cc:407] Connection header received
0925 15:20:41.094418 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:41.094419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:41.094431 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:41.094465 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:41.096009 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.096722 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.097760 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.097983 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.098049 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:41.101902 (+ 3853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:41.101904 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:41.101904 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:41.101912 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:41.103627 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:41.104041 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:41.104042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:41.104042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:41.104131 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:41.104381 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:41.104382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:41.104383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:41.104523 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:41.104569 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:41.104821 (+ 252us) server_negotiation.cc:299] Negotiation successful
0925 15:20:41.104905 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 15:20:41.105041 10211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:41.093507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:46459 (local address 127.3.211.254:38373)
0925 15:20:41.093552 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:41.093554 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:41.093986 (+ 432us) server_negotiation.cc:407] Connection header received
0925 15:20:41.094418 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:41.094419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:41.094432 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:41.094465 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:41.095371 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.095885 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.097082 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.097316 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.097401 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:41.100954 (+ 3553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:41.100956 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:41.100956 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:41.100967 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:41.103085 (+ 2118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:41.103646 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:41.103648 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:41.103648 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:41.103714 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:41.104070 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:41.104071 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:41.104072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:41.104245 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:41.104321 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:41.104793 (+ 472us) server_negotiation.cc:299] Negotiation successful
0925 15:20:41.104894 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 15:20:41.105216 11124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:41.093437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38373 (local address 127.3.211.194:46459)
0925 15:20:41.093927 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:41.093935 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:41.094302 (+ 367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:41.094582 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:41.094586 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:41.095051 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:41.095265 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:41.095271 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.095999 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.096000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:41.096991 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:41.096993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.097408 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.097410 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:41.097539 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:41.098459 (+ 920us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:41.098470 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:41.100809 (+ 2339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:41.103170 (+ 2361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:41.103181 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:41.103184 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:41.103557 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:41.103773 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:41.103774 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:41.103775 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:41.103959 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:41.104322 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:41.104418 (+ 96us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:41.104609 (+ 191us) client_negotiation.cc:769] Sending connection context
0925 15:20:41.104761 (+ 152us) client_negotiation.cc:240] Negotiation successful
0925 15:20:41.104948 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"mutex_wait_us":27,"thread_start_us":61,"threads_started":1}
I20260925 15:20:41.105643 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.195:34379
--local_ip_for_outbound_sockets=127.3.211.195
--tserver_master_addrs=127.3.211.254:38373
--webserver_port=41227
--webserver_interface=127.3.211.195
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/oryx_127.3.211.195.keytab
--principal=oryx/127.3.211.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:42075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.211.254:38373,127.3.211.253:42795
--tserver_master_addrs=127.3.211.254:38373,127.3.211.253:42795,127.3.211.252:36373 with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:20:41.105880 11123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:41.093307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.253:42795 (local address 127.3.211.194:43431)
0925 15:20:41.093670 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:41.093716 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:41.094302 (+ 586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:41.094582 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:41.094586 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:41.095051 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:41.095265 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:41.095271 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.096828 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.096829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:41.097673 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:41.097675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.098089 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.098089 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:41.098141 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:41.099736 (+ 1595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:41.099743 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:41.101797 (+ 2054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:41.103708 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:41.103709 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:41.103710 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:41.103951 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:41.104194 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:41.104194 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:41.104195 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:41.104266 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:41.104658 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:41.104658 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:41.104693 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 15:20:41.105566 (+ 873us) client_negotiation.cc:240] Negotiation successful
0925 15:20:41.105742 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":7,"thread_start_us":97,"threads_started":1}
I20260925 15:20:41.106684 11121 heartbeater.cc:347] Connected to a master server at 127.3.211.253:42795
I20260925 15:20:41.106803 11121 heartbeater.cc:464] Registering TS with master...
I20260925 15:20:41.107034 11121 heartbeater.cc:511] Master 127.3.211.253:42795 requested a full tablet report, sending...
I20260925 15:20:41.107776 10617 ts_manager.cc:194] Registered new tserver with Master: 7921060cf7c04ccf844a869c26b58c4a (127.3.211.194:44803)
I20260925 15:20:41.108372 11120 heartbeater.cc:347] Connected to a master server at 127.3.211.254:38373
I20260925 15:20:41.108470 11120 heartbeater.cc:464] Registering TS with master...
I20260925 15:20:41.108556 11120 heartbeater.cc:511] Master 127.3.211.254:38373 requested a full tablet report, sending...
I20260925 15:20:41.108832 10018 ts_manager.cc:194] Re-registered known tserver with Master: 7921060cf7c04ccf844a869c26b58c4a (127.3.211.194:44803)
I20260925 15:20:41.109009 10986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:41.093362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:50935 (local address 127.3.211.252:36373)
0925 15:20:41.093413 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:41.093415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:41.093812 (+ 397us) server_negotiation.cc:407] Connection header received
0925 15:20:41.094373 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:41.094374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:41.094387 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:41.094450 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:41.096424 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.097027 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.097972 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.098223 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.098311 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:41.105804 (+ 7493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:41.105806 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:41.105806 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:41.105817 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:41.107676 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:41.108098 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:41.108099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:41.108100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:41.108164 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:41.108416 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:41.108417 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:41.108418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:41.108566 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:41.108628 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:41.108827 (+ 199us) server_negotiation.cc:299] Negotiation successful
0925 15:20:41.108925 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 15:20:41.109277 11125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:41.093307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.252:36373 (local address 127.3.211.194:50935)
0925 15:20:41.093683 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:41.093722 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:41.094261 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:41.094583 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:41.094587 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:41.095046 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:41.095263 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:41.095268 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.097140 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.097141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:41.097897 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:41.097899 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.098315 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.098315 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:41.098370 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:41.098778 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:41.098783 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:41.105638 (+ 6855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:41.107777 (+ 2139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:41.107778 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:41.107779 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:41.108003 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:41.108242 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:41.108243 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:41.108243 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:41.108312 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:41.108656 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:41.108657 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:41.108704 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 15:20:41.108786 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 15:20:41.108883 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":26,"thread_start_us":112,"threads_started":1}
I20260925 15:20:41.109292 10018 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.211.194@KRBTEST.COM'} at 127.3.211.194:46459
I20260925 15:20:41.109726 11119 heartbeater.cc:347] Connected to a master server at 127.3.211.252:36373
I20260925 15:20:41.109792 11119 heartbeater.cc:464] Registering TS with master...
I20260925 15:20:41.110018 11119 heartbeater.cc:511] Master 127.3.211.252:36373 requested a full tablet report, sending...
I20260925 15:20:41.110334 10803 ts_manager.cc:194] Registered new tserver with Master: 7921060cf7c04ccf844a869c26b58c4a (127.3.211.194:44803)
W20260925 15:20:41.191764 11126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:41.191987 11126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:41.192013 11126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:41.193239 11126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:41.193300 11126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:41.193328 11126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.195
I20260925 15:20:41.194897 11126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:42075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/krb5kdc/oryx_127.3.211.195.keytab
--principal=oryx/127.3.211.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.195:34379
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.211.195
--webserver_port=41227
--tserver_master_addrs=127.3.211.254:38373,127.3.211.253:42795,127.3.211.252:36373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.195
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:41.195186 11126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:41.195462 11126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:41.200693 11134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:41.200696 11131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:41.200693 11132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:41.201022 11126 server_base.cc:1034] running on GCE node
I20260925 15:20:41.201244 11126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:41.201534 11126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:41.202720 11126 hybrid_clock.cc:648] HybridClock initialized: now 1790349641202686 us; error 31 us; skew 500 ppm
Sep 25 15:20:41 dist-test-slave-4bk3 krb5kdc[9973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349641, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.211.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:41.205587 11126 init.cc:377] Logged in from keytab as oryx/127.3.211.195@KRBTEST.COM (short username oryx)
I20260925 15:20:41.206144 11126 webserver.cc:460] Webserver started at http://127.3.211.195:41227/ using document root <none> and password file <none>
I20260925 15:20:41.206393 11126 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:41.206476 11126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:41.207887 11126 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:20:41.208580 11141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:41.208748 11126 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:41.208851 11126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-2/wal
uuid: "fbc183e336c74b8a8d68f4624d9b375b"
format_stamp: "Formatted at 2026-09-25 15:20:40 on dist-test-slave-4bk3"
server_key: "06cd2690c772a1743ce2d60cf6bff7f6"
server_key_iv: "9a0115d5a88acfac4a5afdecd2eba7cf"
server_key_version: "encryptionkey@0"
I20260925 15:20:41.209192 11126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:41.242244 10576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:41.239666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.252:36373 (local address 127.0.0.1:50416)
0925 15:20:41.239730 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:41.239743 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:41.239798 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:41.239966 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:41.239968 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:41.239969 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:20:41.240027 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:41.240029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.240715 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.240717 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:41.241427 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:41.241429 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.241997 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.241998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:41.242058 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:41.242062 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:20:41.242119 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 15:20:41.242165 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 15:20:41.242244 10986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:41.239742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50416 (local address 127.3.211.252:36373)
0925 15:20:41.239793 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:41.239795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:41.239807 (+ 12us) server_negotiation.cc:407] Connection header received
0925 15:20:41.239861 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:41.239862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:41.239890 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:41.239963 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:20:41.240096 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.240635 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.241517 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.241900 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.241995 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:41.242070 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:41.242125 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 15:20:41.242161 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 15:20:41.242952 10813 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 6a04d4b30a3b47d0a897ed7f3afa45a3 (127.3.211.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38373 } } peers { permanent_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42795 } attrs { promote: false } } peers { permanent_uuid: "881c1ed0940a4a7483c36869cba574f8" member_type: NON_VOTER last_known_addr { host: "127.3.211.252" port: 36373 } attrs { promote: true } } }
I20260925 15:20:41.243207 11126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:41.244308 11147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38373 } } peers { permanent_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42795 } attrs { promote: false } } peers { permanent_uuid: "881c1ed0940a4a7483c36869cba574f8" member_type: NON_VOTER last_known_addr { host: "127.3.211.252" port: 36373 } attrs { promote: true } } }
I20260925 15:20:41.244448 11147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [sys.catalog]: This master's current role is: LEARNER
I20260925 15:20:41.244586 10842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3860a4600f94b6fa48e2ac45ace80e8. Latest consensus state: current_term: 1 leader_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38373 } } peers { permanent_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42795 } attrs { promote: false } } peers { permanent_uuid: "881c1ed0940a4a7483c36869cba574f8" member_type: NON_VOTER last_known_addr { host: "127.3.211.252" port: 36373 } attrs { promote: true } } }
I20260925 15:20:41.244704 10842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [sys.catalog]: This master's current role is: LEARNER
I20260925 15:20:41.245852 11126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:41.245971 11126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:41.246145 11126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:20:41.246487 11126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:20:41.246521 11126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:41.246543 11126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:20:41.246556 11126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:41.252296 11126 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.195:34379
I20260925 15:20:41.252460 11256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.195:34379 every 8 connection(s)
I20260925 15:20:41.252774 11126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
I20260925 15:20:41.253329 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 11126
Sep 25 15:20:41 dist-test-slave-4bk3 krb5kdc[9973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349641, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.211.195@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM
Sep 25 15:20:41 dist-test-slave-4bk3 krb5kdc[9973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349641, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.211.195@KRBTEST.COM for oryx/127.3.211.252@KRBTEST.COM
Sep 25 15:20:41 dist-test-slave-4bk3 krb5kdc[9973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349641, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.211.195@KRBTEST.COM for oryx/127.3.211.253@KRBTEST.COM
I20260925 15:20:41.269008 10986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:41.253661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:39319 (local address 127.3.211.252:36373)
0925 15:20:41.253756 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:41.253758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:41.254064 (+ 306us) server_negotiation.cc:407] Connection header received
0925 15:20:41.256494 (+ 2430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:41.256495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:41.256506 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:41.256565 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:41.258515 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.259216 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.260395 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.260694 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.260748 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:41.264490 (+ 3742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:41.264492 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:41.264492 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:41.264499 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:41.266711 (+ 2212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:41.267387 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:41.267388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:41.267388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:41.267456 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:41.267954 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:41.267955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:41.267956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:41.268136 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:41.268275 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:41.268756 (+ 481us) server_negotiation.cc:299] Negotiation successful
0925 15:20:41.268840 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 15:20:41.269304 10662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:41.254158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:48739 (local address 127.3.211.253:42795)
0925 15:20:41.254202 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:41.254204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:41.254284 (+ 80us) server_negotiation.cc:407] Connection header received
0925 15:20:41.256386 (+ 2102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:41.256387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:41.256399 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:41.256443 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:41.257673 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.258414 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.260044 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.260471 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.260545 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:41.264862 (+ 4317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:41.264863 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:41.264864 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:41.264871 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:41.267996 (+ 3125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:41.268451 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:41.268452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:41.268453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:41.268512 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:41.268720 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:41.268721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:41.268721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:41.268877 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:41.268919 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:41.269121 (+ 202us) server_negotiation.cc:299] Negotiation successful
0925 15:20:41.269214 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 15:20:41.269488 11262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:41.253605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.252:36373 (local address 127.3.211.195:39319)
0925 15:20:41.253901 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:41.253958 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:41.256178 (+ 2220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:41.256678 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:41.256683 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:41.257264 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:41.257730 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:41.257732 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.259312 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.259313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:41.260270 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:41.260272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.260807 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.260808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:41.260887 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:41.261482 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:41.261487 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:41.264387 (+ 2900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:41.266816 (+ 2429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:41.266827 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:41.266830 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:41.267279 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:41.267530 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:41.267531 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:41.267531 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:41.267852 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:41.268367 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:41.268374 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:41.268572 (+ 198us) client_negotiation.cc:769] Sending connection context
0925 15:20:41.268940 (+ 368us) client_negotiation.cc:240] Negotiation successful
0925 15:20:41.269142 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":271,"thread_start_us":95,"threads_started":1}
I20260925 15:20:41.269968 11266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:41.253963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.253:42795 (local address 127.3.211.195:48739)
0925 15:20:41.254222 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:41.254231 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:41.256179 (+ 1948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:41.256677 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:41.256682 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:41.257254 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:41.257584 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:41.257591 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.258541 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.258542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:41.259916 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:41.259919 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.260683 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.260684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:41.260870 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:41.261608 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:41.261613 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:41.264737 (+ 3124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:41.268108 (+ 3371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:41.268109 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:41.268110 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:41.268359 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:41.268589 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:41.268589 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:41.268590 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:41.268649 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:41.268966 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:41.268967 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:41.269007 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 15:20:41.269404 (+ 397us) client_negotiation.cc:240] Negotiation successful
0925 15:20:41.269529 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":4,"thread_start_us":67,"threads_started":1}
I20260925 15:20:41.270084 11257 heartbeater.cc:347] Connected to a master server at 127.3.211.252:36373
I20260925 15:20:41.270177 11257 heartbeater.cc:464] Registering TS with master...
I20260925 15:20:41.270289 11259 heartbeater.cc:347] Connected to a master server at 127.3.211.253:42795
I20260925 15:20:41.270407 11259 heartbeater.cc:464] Registering TS with master...
I20260925 15:20:41.270443 11257 heartbeater.cc:511] Master 127.3.211.252:36373 requested a full tablet report, sending...
I20260925 15:20:41.270475 11259 heartbeater.cc:511] Master 127.3.211.253:42795 requested a full tablet report, sending...
I20260925 15:20:41.270879 10617 ts_manager.cc:194] Registered new tserver with Master: fbc183e336c74b8a8d68f4624d9b375b (127.3.211.195:34379)
I20260925 15:20:41.270879 10803 ts_manager.cc:194] Registered new tserver with Master: fbc183e336c74b8a8d68f4624d9b375b (127.3.211.195:34379)
I20260925 15:20:41.271075 10211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:41.253609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:47331 (local address 127.3.211.254:38373)
0925 15:20:41.253718 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:41.253721 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:41.254064 (+ 343us) server_negotiation.cc:407] Connection header received
0925 15:20:41.256476 (+ 2412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:41.256478 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:41.256490 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:41.256520 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:41.257979 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.258715 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.260005 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.260361 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.260405 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:41.265520 (+ 5115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:41.265522 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:41.265522 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:41.265530 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:41.268049 (+ 2519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:41.269629 (+ 1580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:41.269630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:41.269630 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:41.269703 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:41.270061 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:41.270062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:41.270063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:41.270213 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:41.270264 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:41.270863 (+ 599us) server_negotiation.cc:299] Negotiation successful
0925 15:20:41.270950 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 15:20:41.271579 11261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:41.253607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38373 (local address 127.3.211.195:47331)
0925 15:20:41.253988 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:41.253998 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:41.256386 (+ 2388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:41.256677 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:41.256682 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:41.257843 (+ 1161us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:41.257893 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:41.257894 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.258806 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.258807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:41.259901 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:41.259904 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.260498 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.260499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:41.260665 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:41.261322 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:41.261334 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:41.265378 (+ 4044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:41.268617 (+ 3239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:41.268617 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:41.268618 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:41.268837 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:41.269795 (+ 958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:41.269796 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:41.269796 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:41.269973 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:41.270663 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:41.270664 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:41.270710 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 15:20:41.271159 (+ 449us) client_negotiation.cc:240] Negotiation successful
0925 15:20:41.271283 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"spinlock_wait_cycles":9984,"thread_start_us":110,"threads_started":1}
I20260925 15:20:41.272246 11258 heartbeater.cc:347] Connected to a master server at 127.3.211.254:38373
I20260925 15:20:41.272310 11258 heartbeater.cc:464] Registering TS with master...
I20260925 15:20:41.272392 11268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:41.254585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50422 (local address 127.3.211.252:36373)
0925 15:20:41.255118 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:41.255120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:41.255185 (+ 65us) server_negotiation.cc:407] Connection header received
0925 15:20:41.255430 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:41.255431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:41.255445 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:41.255502 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:41.257430 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.261639 (+ 4209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.263358 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.263673 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.263745 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:41.266658 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:41.266660 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:41.266661 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:41.266669 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:41.268494 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:41.268883 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:41.268884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:41.268884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:41.271531 (+ 2647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:41.271787 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:41.271788 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:41.271789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:41.271925 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:41.272037 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:41.272150 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 15:20:41.272239 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":70,"threads_started":1}
I20260925 15:20:41.272435 11258 heartbeater.cc:511] Master 127.3.211.254:38373 requested a full tablet report, sending...
I20260925 15:20:41.272673 10018 ts_manager.cc:194] Re-registered known tserver with Master: fbc183e336c74b8a8d68f4624d9b375b (127.3.211.195:34379)
I20260925 15:20:41.273047 10018 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.211.195@KRBTEST.COM'} at 127.3.211.195:47331
I20260925 15:20:41.274464 11271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:41.254833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52962 (local address 127.3.211.253:42795)
0925 15:20:41.255348 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:41.255350 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:41.255357 (+ 7us) server_negotiation.cc:407] Connection header received
0925 15:20:41.255717 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:41.255718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:41.255731 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:41.255772 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:41.257331 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.258074 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.263948 (+ 5874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.264829 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.264881 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:41.270233 (+ 5352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:41.270234 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:41.270235 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:41.270243 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:41.271919 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:41.273350 (+ 1431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:41.273351 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:41.273352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:41.273422 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:41.273645 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:41.273646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:41.273647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:41.273787 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:41.273871 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:41.274110 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 15:20:41.274332 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":48,"threads_started":1}
I20260925 15:20:41.275054 11273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:41.255034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47614 (local address 127.3.211.254:38373)
0925 15:20:41.255416 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:41.255418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:41.255424 (+ 6us) server_negotiation.cc:407] Connection header received
0925 15:20:41.255448 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:41.255448 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:41.255460 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:41.255580 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:41.256409 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.257194 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.259078 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.263763 (+ 4685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.263849 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:41.265774 (+ 1925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:41.265775 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:41.265775 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:41.265781 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:41.270120 (+ 4339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:41.272582 (+ 2462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:41.272583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:41.272583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:41.273211 (+ 628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:41.273434 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:41.273435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:41.273436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:41.273567 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:41.273667 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:41.274858 (+ 1191us) server_negotiation.cc:299] Negotiation successful
0925 15:20:41.274948 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":81,"threads_started":1}
I20260925 15:20:41.276962 10018 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47614:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:20:41.278559 10018 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:20:41.279395 10578 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38373 } } peers { permanent_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42795 } attrs { promote: false } } peers { permanent_uuid: "881c1ed0940a4a7483c36869cba574f8" member_type: NON_VOTER last_known_addr { host: "127.3.211.252" port: 36373 } attrs { promote: true } } }
I20260925 15:20:41.279568 10578 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8 [sys.catalog]: This master's current role is: LEADER
I20260925 15:20:41.281317 10578 raft_consensus.cc:1050] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8: attempting to promote NON_VOTER 881c1ed0940a4a7483c36869cba574f8 to VOTER
I20260925 15:20:41.281474 10578 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38373 } } peers { permanent_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42795 } attrs { promote: false } } peers { permanent_uuid: "881c1ed0940a4a7483c36869cba574f8" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 36373 } attrs { promote: false } }
I20260925 15:20:41.281867 10627 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3 [term 1 FOLLOWER]: Refusing update from remote peer b3860a4600f94b6fa48e2ac45ace80e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260925 15:20:41.281913 10813 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [term 1 LEARNER]: Refusing update from remote peer b3860a4600f94b6fa48e2ac45ace80e8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260925 15:20:41.282078 10775 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42795 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260925 15:20:41.282158 10578 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "881c1ed0940a4a7483c36869cba574f8" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 36373 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260925 15:20:41.282909 10578 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 881c1ed0940a4a7483c36869cba574f8 (127.3.211.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38373 } } peers { permanent_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42795 } attrs { promote: false } } peers { permanent_uuid: "881c1ed0940a4a7483c36869cba574f8" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 36373 } attrs { promote: false } } }
I20260925 15:20:41.283082 10627 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 881c1ed0940a4a7483c36869cba574f8 (127.3.211.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38373 } } peers { permanent_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42795 } attrs { promote: false } } peers { permanent_uuid: "881c1ed0940a4a7483c36869cba574f8" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 36373 } attrs { promote: false } } }
I20260925 15:20:41.283113 10813 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 881c1ed0940a4a7483c36869cba574f8 (127.3.211.252) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38373 } } peers { permanent_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42795 } attrs { promote: false } } peers { permanent_uuid: "881c1ed0940a4a7483c36869cba574f8" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 36373 } attrs { promote: false } } }
I20260925 15:20:41.283911 10578 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38373 } } peers { permanent_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42795 } attrs { promote: false } } peers { permanent_uuid: "881c1ed0940a4a7483c36869cba574f8" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 36373 } attrs { promote: false } } }
I20260925 15:20:41.284001 10578 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8 [sys.catalog]: This master's current role is: LEADER
I20260925 15:20:41.284008 10768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38373 } } peers { permanent_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42795 } attrs { promote: false } } peers { permanent_uuid: "881c1ed0940a4a7483c36869cba574f8" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 36373 } attrs { promote: false } } }
I20260925 15:20:41.284096 10768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:20:41.284067 10578 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38373 } } peers { permanent_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42795 } attrs { promote: false } } peers { permanent_uuid: "881c1ed0940a4a7483c36869cba574f8" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 36373 } attrs { promote: false } } }
I20260925 15:20:41.284075 10842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38373 } } peers { permanent_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42795 } attrs { promote: false } } peers { permanent_uuid: "881c1ed0940a4a7483c36869cba574f8" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 36373 } attrs { promote: false } } }
I20260925 15:20:41.284133 10578 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8 [sys.catalog]: This master's current role is: LEADER
I20260925 15:20:41.284144 10842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:20:41.284179 10578 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38373 } } peers { permanent_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42795 } attrs { promote: false } } peers { permanent_uuid: "881c1ed0940a4a7483c36869cba574f8" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 36373 } attrs { promote: false } } }
I20260925 15:20:41.284233 10578 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8 [sys.catalog]: This master's current role is: LEADER
I20260925 15:20:41.289654 11284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:41.285452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38596 (local address 127.3.211.193:32847)
0925 15:20:41.285602 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:41.285606 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:41.285616 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:20:41.285651 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:41.285652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:41.285698 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:41.285794 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:20:41.285870 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.287211 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.288431 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.289300 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.289366 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:41.289433 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:41.289508 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 15:20:41.289557 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":69,"threads_started":1}
I20260925 15:20:41.289641 10576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:41.285123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:32847 (local address 127.0.0.1:38596)
0925 15:20:41.285335 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:41.285343 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:41.285374 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:41.285789 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:41.285789 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:41.285790 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:20:41.285817 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:41.285818 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.287437 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.287438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:41.288322 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:41.288325 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.289363 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.289364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:41.289427 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:41.289431 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:20:41.289488 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 15:20:41.289538 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":4,"thread_start_us":97,"threads_started":1}
I20260925 15:20:41.289949 11283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:41.285320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43922 (local address 127.3.211.194:44803)
0925 15:20:41.285464 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:41.285469 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:41.285482 (+ 13us) server_negotiation.cc:407] Connection header received
0925 15:20:41.285527 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:41.285529 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:41.285550 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:41.285651 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:20:41.285901 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.286792 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.288171 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.289246 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.289330 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:41.289410 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:41.289682 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 15:20:41.289736 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":77,"threads_started":1}
I20260925 15:20:41.290107 11282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:41.285238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46910 (local address 127.3.211.195:34379)
0925 15:20:41.285506 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:41.285510 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:41.285676 (+ 166us) server_negotiation.cc:407] Connection header received
0925 15:20:41.285770 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:41.285772 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:41.285789 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:41.285865 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:20:41.285931 (+ 66us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.287084 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.288315 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.289146 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.289204 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:41.289265 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:41.289901 (+ 636us) server_negotiation.cc:299] Negotiation successful
0925 15:20:41.289958 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":75,"threads_started":1}
I20260925 15:20:41.289649 11280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:41.285094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.194:44803 (local address 127.0.0.1:43922)
0925 15:20:41.285265 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:41.285277 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:41.285335 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:41.285661 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:41.285662 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:41.285664 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:20:41.285727 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:41.285729 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.286873 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.286874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:41.288067 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:41.288070 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.289353 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.289354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:41.289430 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:41.289433 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:20:41.289493 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 15:20:41.289538 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145}
I20260925 15:20:41.290460 11281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:41.285146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.195:34379 (local address 127.0.0.1:46910)
0925 15:20:41.285599 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:41.285607 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:41.285682 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:41.285856 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:41.285857 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:41.285858 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:20:41.285890 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:41.285891 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.287150 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.287151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:41.288227 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:41.288229 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.289472 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.289474 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:41.289541 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:41.289546 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:20:41.289600 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 15:20:41.289659 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":108,"threads_started":1}
I20260925 15:20:41.290597 10916 tablet_service.cc:1467] Processing CreateTablet for tablet ecd9c6c9a2b74a1c8b104fa3affff363 (DEFAULT_TABLE table=test-table [id=806ac27304014d6186c4d079f91aff7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:20:41.290977 10916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecd9c6c9a2b74a1c8b104fa3affff363. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:41.290992 11053 tablet_service.cc:1467] Processing CreateTablet for tablet ecd9c6c9a2b74a1c8b104fa3affff363 (DEFAULT_TABLE table=test-table [id=806ac27304014d6186c4d079f91aff7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:20:41.291081 11191 tablet_service.cc:1467] Processing CreateTablet for tablet ecd9c6c9a2b74a1c8b104fa3affff363 (DEFAULT_TABLE table=test-table [id=806ac27304014d6186c4d079f91aff7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:20:41.291306 11053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecd9c6c9a2b74a1c8b104fa3affff363. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:41.291428 11191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecd9c6c9a2b74a1c8b104fa3affff363. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:41.294353 11285 tablet_bootstrap.cc:492] T ecd9c6c9a2b74a1c8b104fa3affff363 P fbc183e336c74b8a8d68f4624d9b375b: Bootstrap starting.
I20260925 15:20:41.295015 11287 tablet_bootstrap.cc:492] T ecd9c6c9a2b74a1c8b104fa3affff363 P bf6c0d3ea9e342d6960a6f8a5d1e7e26: Bootstrap starting.
I20260925 15:20:41.295197 11285 tablet_bootstrap.cc:654] T ecd9c6c9a2b74a1c8b104fa3affff363 P fbc183e336c74b8a8d68f4624d9b375b: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:41.295553 11285 log.cc:824] T ecd9c6c9a2b74a1c8b104fa3affff363 P fbc183e336c74b8a8d68f4624d9b375b: Log is configured to *not* fsync() on all Append() calls
I20260925 15:20:41.295672 11287 tablet_bootstrap.cc:654] T ecd9c6c9a2b74a1c8b104fa3affff363 P bf6c0d3ea9e342d6960a6f8a5d1e7e26: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:41.295950 11287 log.cc:824] T ecd9c6c9a2b74a1c8b104fa3affff363 P bf6c0d3ea9e342d6960a6f8a5d1e7e26: Log is configured to *not* fsync() on all Append() calls
I20260925 15:20:41.296283 11286 tablet_bootstrap.cc:492] T ecd9c6c9a2b74a1c8b104fa3affff363 P 7921060cf7c04ccf844a869c26b58c4a: Bootstrap starting.
I20260925 15:20:41.296386 11285 tablet_bootstrap.cc:492] T ecd9c6c9a2b74a1c8b104fa3affff363 P fbc183e336c74b8a8d68f4624d9b375b: No bootstrap required, opened a new log
I20260925 15:20:41.296453 11285 ts_tablet_manager.cc:1397] T ecd9c6c9a2b74a1c8b104fa3affff363 P fbc183e336c74b8a8d68f4624d9b375b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:20:41.296762 11287 tablet_bootstrap.cc:492] T ecd9c6c9a2b74a1c8b104fa3affff363 P bf6c0d3ea9e342d6960a6f8a5d1e7e26: No bootstrap required, opened a new log
I20260925 15:20:41.296850 11287 ts_tablet_manager.cc:1397] T ecd9c6c9a2b74a1c8b104fa3affff363 P bf6c0d3ea9e342d6960a6f8a5d1e7e26: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:20:41.296943 11286 tablet_bootstrap.cc:654] T ecd9c6c9a2b74a1c8b104fa3affff363 P 7921060cf7c04ccf844a869c26b58c4a: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:41.297336 11286 log.cc:824] T ecd9c6c9a2b74a1c8b104fa3affff363 P 7921060cf7c04ccf844a869c26b58c4a: Log is configured to *not* fsync() on all Append() calls
I20260925 15:20:41.298324 11286 tablet_bootstrap.cc:492] T ecd9c6c9a2b74a1c8b104fa3affff363 P 7921060cf7c04ccf844a869c26b58c4a: No bootstrap required, opened a new log
I20260925 15:20:41.298406 11286 ts_tablet_manager.cc:1397] T ecd9c6c9a2b74a1c8b104fa3affff363 P 7921060cf7c04ccf844a869c26b58c4a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:20:41.299242 11287 raft_consensus.cc:348] T ecd9c6c9a2b74a1c8b104fa3affff363 P bf6c0d3ea9e342d6960a6f8a5d1e7e26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7921060cf7c04ccf844a869c26b58c4a" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44803 } } peers { permanent_uuid: "bf6c0d3ea9e342d6960a6f8a5d1e7e26" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 32847 } } peers { permanent_uuid: "fbc183e336c74b8a8d68f4624d9b375b" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 34379 } }
I20260925 15:20:41.299417 11287 raft_consensus.cc:374] T ecd9c6c9a2b74a1c8b104fa3affff363 P bf6c0d3ea9e342d6960a6f8a5d1e7e26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:41.299459 11287 raft_consensus.cc:729] T ecd9c6c9a2b74a1c8b104fa3affff363 P bf6c0d3ea9e342d6960a6f8a5d1e7e26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf6c0d3ea9e342d6960a6f8a5d1e7e26, State: Initialized, Role: FOLLOWER
I20260925 15:20:41.299580 11287 consensus_queue.cc:260] T ecd9c6c9a2b74a1c8b104fa3affff363 P bf6c0d3ea9e342d6960a6f8a5d1e7e26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7921060cf7c04ccf844a869c26b58c4a" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44803 } } peers { permanent_uuid: "bf6c0d3ea9e342d6960a6f8a5d1e7e26" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 32847 } } peers { permanent_uuid: "fbc183e336c74b8a8d68f4624d9b375b" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 34379 } }
I20260925 15:20:41.299870 11287 ts_tablet_manager.cc:1428] T ecd9c6c9a2b74a1c8b104fa3affff363 P bf6c0d3ea9e342d6960a6f8a5d1e7e26: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:20:41.300091 10983 heartbeater.cc:503] Master 127.3.211.254:38373 was elected leader, sending a full tablet report...
I20260925 15:20:41.300918 11286 raft_consensus.cc:348] T ecd9c6c9a2b74a1c8b104fa3affff363 P 7921060cf7c04ccf844a869c26b58c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7921060cf7c04ccf844a869c26b58c4a" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44803 } } peers { permanent_uuid: "bf6c0d3ea9e342d6960a6f8a5d1e7e26" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 32847 } } peers { permanent_uuid: "fbc183e336c74b8a8d68f4624d9b375b" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 34379 } }
I20260925 15:20:41.299716 11285 raft_consensus.cc:348] T ecd9c6c9a2b74a1c8b104fa3affff363 P fbc183e336c74b8a8d68f4624d9b375b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7921060cf7c04ccf844a869c26b58c4a" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44803 } } peers { permanent_uuid: "bf6c0d3ea9e342d6960a6f8a5d1e7e26" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 32847 } } peers { permanent_uuid: "fbc183e336c74b8a8d68f4624d9b375b" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 34379 } }
I20260925 15:20:41.301090 11286 raft_consensus.cc:374] T ecd9c6c9a2b74a1c8b104fa3affff363 P 7921060cf7c04ccf844a869c26b58c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:41.301093 11285 raft_consensus.cc:374] T ecd9c6c9a2b74a1c8b104fa3affff363 P fbc183e336c74b8a8d68f4624d9b375b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:41.301131 11286 raft_consensus.cc:729] T ecd9c6c9a2b74a1c8b104fa3affff363 P 7921060cf7c04ccf844a869c26b58c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7921060cf7c04ccf844a869c26b58c4a, State: Initialized, Role: FOLLOWER
I20260925 15:20:41.301136 11285 raft_consensus.cc:729] T ecd9c6c9a2b74a1c8b104fa3affff363 P fbc183e336c74b8a8d68f4624d9b375b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbc183e336c74b8a8d68f4624d9b375b, State: Initialized, Role: FOLLOWER
I20260925 15:20:41.301272 11285 consensus_queue.cc:260] T ecd9c6c9a2b74a1c8b104fa3affff363 P fbc183e336c74b8a8d68f4624d9b375b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7921060cf7c04ccf844a869c26b58c4a" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44803 } } peers { permanent_uuid: "bf6c0d3ea9e342d6960a6f8a5d1e7e26" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 32847 } } peers { permanent_uuid: "fbc183e336c74b8a8d68f4624d9b375b" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 34379 } }
I20260925 15:20:41.301280 11286 consensus_queue.cc:260] T ecd9c6c9a2b74a1c8b104fa3affff363 P 7921060cf7c04ccf844a869c26b58c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7921060cf7c04ccf844a869c26b58c4a" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44803 } } peers { permanent_uuid: "bf6c0d3ea9e342d6960a6f8a5d1e7e26" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 32847 } } peers { permanent_uuid: "fbc183e336c74b8a8d68f4624d9b375b" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 34379 } }
I20260925 15:20:41.301602 11286 ts_tablet_manager.cc:1428] T ecd9c6c9a2b74a1c8b104fa3affff363 P 7921060cf7c04ccf844a869c26b58c4a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:20:41.301594 11285 ts_tablet_manager.cc:1428] T ecd9c6c9a2b74a1c8b104fa3affff363 P fbc183e336c74b8a8d68f4624d9b375b: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260925 15:20:41.301693 11258 heartbeater.cc:503] Master 127.3.211.254:38373 was elected leader, sending a full tablet report...
I20260925 15:20:41.302527 11120 heartbeater.cc:503] Master 127.3.211.254:38373 was elected leader, sending a full tablet report...
W20260925 15:20:41.344962 11122 tablet.cc:2369] T ecd9c6c9a2b74a1c8b104fa3affff363 P 7921060cf7c04ccf844a869c26b58c4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:20:41.430467 10985 tablet.cc:2369] T ecd9c6c9a2b74a1c8b104fa3affff363 P bf6c0d3ea9e342d6960a6f8a5d1e7e26: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:20:41.504693 11260 tablet.cc:2369] T ecd9c6c9a2b74a1c8b104fa3affff363 P fbc183e336c74b8a8d68f4624d9b375b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:20:41.507295 11291 raft_consensus.cc:482] T ecd9c6c9a2b74a1c8b104fa3affff363 P bf6c0d3ea9e342d6960a6f8a5d1e7e26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:20:41.507454 11291 raft_consensus.cc:504] T ecd9c6c9a2b74a1c8b104fa3affff363 P bf6c0d3ea9e342d6960a6f8a5d1e7e26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7921060cf7c04ccf844a869c26b58c4a" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44803 } } peers { permanent_uuid: "bf6c0d3ea9e342d6960a6f8a5d1e7e26" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 32847 } } peers { permanent_uuid: "fbc183e336c74b8a8d68f4624d9b375b" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 34379 } }
I20260925 15:20:41.507863 11291 leader_election.cc:288] T ecd9c6c9a2b74a1c8b104fa3affff363 P bf6c0d3ea9e342d6960a6f8a5d1e7e26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7921060cf7c04ccf844a869c26b58c4a (127.3.211.194:44803), fbc183e336c74b8a8d68f4624d9b375b (127.3.211.195:34379)
I20260925 15:20:41.512354 11282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:41.508164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:48591 (local address 127.3.211.195:34379)
0925 15:20:41.508233 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:41.508235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:41.508293 (+ 58us) server_negotiation.cc:407] Connection header received
0925 15:20:41.508415 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:41.508416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:41.508426 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:41.508495 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:20:41.508735 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.509503 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.511224 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.511846 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.511941 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:41.511952 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:41.512206 (+ 254us) server_negotiation.cc:299] Negotiation successful
0925 15:20:41.512255 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 15:20:41.512355 11283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:41.508214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:48405 (local address 127.3.211.194:44803)
0925 15:20:41.508257 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:41.508259 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:41.508293 (+ 34us) server_negotiation.cc:407] Connection header received
0925 15:20:41.508418 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:41.508419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:41.508427 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:41.508495 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:20:41.508736 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.509503 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.511224 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.511848 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.511941 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:41.511952 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:41.512180 (+ 228us) server_negotiation.cc:299] Negotiation successful
0925 15:20:41.512249 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 15:20:41.512400 11294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:41.508079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.194:44803 (local address 127.3.211.193:48405)
0925 15:20:41.508224 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:41.508235 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:41.508311 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:41.508511 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:41.508512 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:41.508515 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:20:41.508613 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:41.508616 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.509654 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.509655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:41.511085 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:41.511089 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.511962 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.511963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:41.512071 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:41.512079 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 15:20:41.512160 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 15:20:41.512220 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":65,"thread_start_us":79,"threads_started":1}
I20260925 15:20:41.512414 11295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:41.508080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.195:34379 (local address 127.3.211.193:48591)
0925 15:20:41.508234 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:41.508243 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:41.508311 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:41.508511 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:41.508512 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:41.508514 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:20:41.508642 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:41.508644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.509641 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.509644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:41.511085 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:41.511089 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.511962 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.511963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:41.512071 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:41.512078 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 15:20:41.512160 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 15:20:41.512220 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":123,"thread_start_us":72,"threads_started":1}
I20260925 15:20:41.513137 11211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecd9c6c9a2b74a1c8b104fa3affff363" candidate_uuid: "bf6c0d3ea9e342d6960a6f8a5d1e7e26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbc183e336c74b8a8d68f4624d9b375b" is_pre_election: true
I20260925 15:20:41.513283 11073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecd9c6c9a2b74a1c8b104fa3affff363" candidate_uuid: "bf6c0d3ea9e342d6960a6f8a5d1e7e26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7921060cf7c04ccf844a869c26b58c4a" is_pre_election: true
I20260925 15:20:41.513334 11211 raft_consensus.cc:2454] T ecd9c6c9a2b74a1c8b104fa3affff363 P fbc183e336c74b8a8d68f4624d9b375b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf6c0d3ea9e342d6960a6f8a5d1e7e26 in term 0.
I20260925 15:20:41.513437 11073 raft_consensus.cc:2454] T ecd9c6c9a2b74a1c8b104fa3affff363 P 7921060cf7c04ccf844a869c26b58c4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf6c0d3ea9e342d6960a6f8a5d1e7e26 in term 0.
I20260925 15:20:41.513579 10872 leader_election.cc:302] T ecd9c6c9a2b74a1c8b104fa3affff363 P bf6c0d3ea9e342d6960a6f8a5d1e7e26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf6c0d3ea9e342d6960a6f8a5d1e7e26, fbc183e336c74b8a8d68f4624d9b375b; no voters:
I20260925 15:20:41.513777 11291 raft_consensus.cc:2781] T ecd9c6c9a2b74a1c8b104fa3affff363 P bf6c0d3ea9e342d6960a6f8a5d1e7e26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:20:41.513829 11291 raft_consensus.cc:482] T ecd9c6c9a2b74a1c8b104fa3affff363 P bf6c0d3ea9e342d6960a6f8a5d1e7e26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:20:41.513849 11291 raft_consensus.cc:3037] T ecd9c6c9a2b74a1c8b104fa3affff363 P bf6c0d3ea9e342d6960a6f8a5d1e7e26 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:41.514616 11291 raft_consensus.cc:504] T ecd9c6c9a2b74a1c8b104fa3affff363 P bf6c0d3ea9e342d6960a6f8a5d1e7e26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7921060cf7c04ccf844a869c26b58c4a" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44803 } } peers { permanent_uuid: "bf6c0d3ea9e342d6960a6f8a5d1e7e26" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 32847 } } peers { permanent_uuid: "fbc183e336c74b8a8d68f4624d9b375b" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 34379 } }
I20260925 15:20:41.514775 11291 leader_election.cc:288] T ecd9c6c9a2b74a1c8b104fa3affff363 P bf6c0d3ea9e342d6960a6f8a5d1e7e26 [CANDIDATE]: Term 1 election: Requested vote from peers 7921060cf7c04ccf844a869c26b58c4a (127.3.211.194:44803), fbc183e336c74b8a8d68f4624d9b375b (127.3.211.195:34379)
I20260925 15:20:41.514918 11073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecd9c6c9a2b74a1c8b104fa3affff363" candidate_uuid: "bf6c0d3ea9e342d6960a6f8a5d1e7e26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7921060cf7c04ccf844a869c26b58c4a"
I20260925 15:20:41.514946 11211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecd9c6c9a2b74a1c8b104fa3affff363" candidate_uuid: "bf6c0d3ea9e342d6960a6f8a5d1e7e26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbc183e336c74b8a8d68f4624d9b375b"
I20260925 15:20:41.515003 11073 raft_consensus.cc:3037] T ecd9c6c9a2b74a1c8b104fa3affff363 P 7921060cf7c04ccf844a869c26b58c4a [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:41.515019 11211 raft_consensus.cc:3037] T ecd9c6c9a2b74a1c8b104fa3affff363 P fbc183e336c74b8a8d68f4624d9b375b [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:41.515916 11073 raft_consensus.cc:2454] T ecd9c6c9a2b74a1c8b104fa3affff363 P 7921060cf7c04ccf844a869c26b58c4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf6c0d3ea9e342d6960a6f8a5d1e7e26 in term 1.
I20260925 15:20:41.515918 11211 raft_consensus.cc:2454] T ecd9c6c9a2b74a1c8b104fa3affff363 P fbc183e336c74b8a8d68f4624d9b375b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf6c0d3ea9e342d6960a6f8a5d1e7e26 in term 1.
I20260925 15:20:41.516108 10872 leader_election.cc:302] T ecd9c6c9a2b74a1c8b104fa3affff363 P bf6c0d3ea9e342d6960a6f8a5d1e7e26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf6c0d3ea9e342d6960a6f8a5d1e7e26, fbc183e336c74b8a8d68f4624d9b375b; no voters:
I20260925 15:20:41.516213 11291 raft_consensus.cc:2781] T ecd9c6c9a2b74a1c8b104fa3affff363 P bf6c0d3ea9e342d6960a6f8a5d1e7e26 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:20:41.516371 11291 raft_consensus.cc:686] T ecd9c6c9a2b74a1c8b104fa3affff363 P bf6c0d3ea9e342d6960a6f8a5d1e7e26 [term 1 LEADER]: Becoming Leader. State: Replica: bf6c0d3ea9e342d6960a6f8a5d1e7e26, State: Running, Role: LEADER
I20260925 15:20:41.516471 11291 consensus_queue.cc:237] T ecd9c6c9a2b74a1c8b104fa3affff363 P bf6c0d3ea9e342d6960a6f8a5d1e7e26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7921060cf7c04ccf844a869c26b58c4a" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44803 } } peers { permanent_uuid: "bf6c0d3ea9e342d6960a6f8a5d1e7e26" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 32847 } } peers { permanent_uuid: "fbc183e336c74b8a8d68f4624d9b375b" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 34379 } }
I20260925 15:20:41.517261 10018 catalog_manager.cc:5495] T ecd9c6c9a2b74a1c8b104fa3affff363 P bf6c0d3ea9e342d6960a6f8a5d1e7e26 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf6c0d3ea9e342d6960a6f8a5d1e7e26 (127.3.211.193). New cstate: current_term: 1 leader_uuid: "bf6c0d3ea9e342d6960a6f8a5d1e7e26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7921060cf7c04ccf844a869c26b58c4a" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf6c0d3ea9e342d6960a6f8a5d1e7e26" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 32847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fbc183e336c74b8a8d68f4624d9b375b" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 34379 } health_report { overall_health: UNKNOWN } } }
I20260925 15:20:41.574220 11284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:41.571784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38612 (local address 127.3.211.193:32847)
0925 15:20:41.571867 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:41.571869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:41.571878 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:20:41.571918 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:41.571919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:41.571941 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:41.572017 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:20:41.572151 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.572653 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.573426 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.573632 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.573719 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:41.573879 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:20:41.573996 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:20:41.574092 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:41.574110 (+ 18us) server_negotiation.cc:299] Negotiation successful
0925 15:20:41.574131 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 15:20:41.576283 11073 raft_consensus.cc:1261] T ecd9c6c9a2b74a1c8b104fa3affff363 P 7921060cf7c04ccf844a869c26b58c4a [term 1 FOLLOWER]: Refusing update from remote peer bf6c0d3ea9e342d6960a6f8a5d1e7e26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:20:41.576283 11211 raft_consensus.cc:1261] T ecd9c6c9a2b74a1c8b104fa3affff363 P fbc183e336c74b8a8d68f4624d9b375b [term 1 FOLLOWER]: Refusing update from remote peer bf6c0d3ea9e342d6960a6f8a5d1e7e26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:20:41.576730 11296 consensus_queue.cc:1035] T ecd9c6c9a2b74a1c8b104fa3affff363 P bf6c0d3ea9e342d6960a6f8a5d1e7e26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbc183e336c74b8a8d68f4624d9b375b" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 34379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:20:41.576922 11291 consensus_queue.cc:1035] T ecd9c6c9a2b74a1c8b104fa3affff363 P bf6c0d3ea9e342d6960a6f8a5d1e7e26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7921060cf7c04ccf844a869c26b58c4a" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:20:41.579312 11299 mvcc.cc:205] Tried to move back new op lower bound from 7333272131892932608 to 7333272131652005888. Current Snapshot: MvccSnapshot[applied={T|T < 7333272131892932608}]
I20260925 15:20:41.580561 11301 mvcc.cc:205] Tried to move back new op lower bound from 7333272131892932608 to 7333272131652005888. Current Snapshot: MvccSnapshot[applied={T|T < 7333272131892932608 or (T in {7333272131892932608})}]
I20260925 15:20:41.580875 11300 mvcc.cc:205] Tried to move back new op lower bound from 7333272131892932608 to 7333272131652005888. Current Snapshot: MvccSnapshot[applied={T|T < 7333272131892932608}]
I20260925 15:20:41.583961 10018 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:20:41.584084 10018 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:20:41.585423 10018 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b3860a4600f94b6fa48e2ac45ace80e8: Sending DeleteTablet for 3 replicas of tablet ecd9c6c9a2b74a1c8b104fa3affff363
I20260925 15:20:41.585785 10916 tablet_service.cc:1514] Processing DeleteTablet for tablet ecd9c6c9a2b74a1c8b104fa3affff363 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:20:41 UTC) from {username='oryx', principal='oryx/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:38596
I20260925 15:20:41.585809 11053 tablet_service.cc:1514] Processing DeleteTablet for tablet ecd9c6c9a2b74a1c8b104fa3affff363 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:20:41 UTC) from {username='oryx', principal='oryx/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:43922
I20260925 15:20:41.586087 11310 tablet_replica.cc:330] T ecd9c6c9a2b74a1c8b104fa3affff363 P bf6c0d3ea9e342d6960a6f8a5d1e7e26: stopping tablet replica
I20260925 15:20:41.586222 11310 raft_consensus.cc:2229] T ecd9c6c9a2b74a1c8b104fa3affff363 P bf6c0d3ea9e342d6960a6f8a5d1e7e26 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:20:41.586294 11191 tablet_service.cc:1514] Processing DeleteTablet for tablet ecd9c6c9a2b74a1c8b104fa3affff363 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:20:41 UTC) from {username='oryx', principal='oryx/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:46910
I20260925 15:20:41.586344 11310 raft_consensus.cc:2258] T ecd9c6c9a2b74a1c8b104fa3affff363 P bf6c0d3ea9e342d6960a6f8a5d1e7e26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:20:41.586508 11311 tablet_replica.cc:330] T ecd9c6c9a2b74a1c8b104fa3affff363 P 7921060cf7c04ccf844a869c26b58c4a: stopping tablet replica
I20260925 15:20:41.586621 11311 raft_consensus.cc:2229] T ecd9c6c9a2b74a1c8b104fa3affff363 P 7921060cf7c04ccf844a869c26b58c4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:20:41.586736 11311 raft_consensus.cc:2258] T ecd9c6c9a2b74a1c8b104fa3affff363 P 7921060cf7c04ccf844a869c26b58c4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:20:41.586879 11310 ts_tablet_manager.cc:1905] T ecd9c6c9a2b74a1c8b104fa3affff363 P bf6c0d3ea9e342d6960a6f8a5d1e7e26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:20:41.587172 11311 ts_tablet_manager.cc:1905] T ecd9c6c9a2b74a1c8b104fa3affff363 P 7921060cf7c04ccf844a869c26b58c4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:20:41.587494 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 9986
I20260925 15:20:41.588938 11311 ts_tablet_manager.cc:1918] T ecd9c6c9a2b74a1c8b104fa3affff363 P 7921060cf7c04ccf844a869c26b58c4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:20:41.589025 11311 log.cc:1196] T ecd9c6c9a2b74a1c8b104fa3affff363 P 7921060cf7c04ccf844a869c26b58c4a: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-1/wal/wals/ecd9c6c9a2b74a1c8b104fa3affff363
I20260925 15:20:41.589243 11310 ts_tablet_manager.cc:1918] T ecd9c6c9a2b74a1c8b104fa3affff363 P bf6c0d3ea9e342d6960a6f8a5d1e7e26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:20:41.589315 11311 ts_tablet_manager.cc:1939] T ecd9c6c9a2b74a1c8b104fa3affff363 P 7921060cf7c04ccf844a869c26b58c4a: Deleting consensus metadata
I20260925 15:20:41.589336 11310 log.cc:1196] T ecd9c6c9a2b74a1c8b104fa3affff363 P bf6c0d3ea9e342d6960a6f8a5d1e7e26: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-0/wal/wals/ecd9c6c9a2b74a1c8b104fa3affff363
I20260925 15:20:41.589587 11310 ts_tablet_manager.cc:1939] T ecd9c6c9a2b74a1c8b104fa3affff363 P bf6c0d3ea9e342d6960a6f8a5d1e7e26: Deleting consensus metadata
I20260925 15:20:41.592789 11312 tablet_replica.cc:330] T ecd9c6c9a2b74a1c8b104fa3affff363 P fbc183e336c74b8a8d68f4624d9b375b: stopping tablet replica
I20260925 15:20:41.592959 11312 raft_consensus.cc:2229] T ecd9c6c9a2b74a1c8b104fa3affff363 P fbc183e336c74b8a8d68f4624d9b375b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:20:41.593091 11312 raft_consensus.cc:2258] T ecd9c6c9a2b74a1c8b104fa3affff363 P fbc183e336c74b8a8d68f4624d9b375b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:20:41.593613 11312 ts_tablet_manager.cc:1905] T ecd9c6c9a2b74a1c8b104fa3affff363 P fbc183e336c74b8a8d68f4624d9b375b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 15:20:41.594113 10871 connection.cc:535] server connection from 127.0.0.1:38596 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260925 15:20:41.594342 11007 connection.cc:535] server connection from 127.0.0.1:43922 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 15:20:41.595384 11312 ts_tablet_manager.cc:1918] T ecd9c6c9a2b74a1c8b104fa3affff363 P fbc183e336c74b8a8d68f4624d9b375b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:20:41.595471 11312 log.cc:1196] T ecd9c6c9a2b74a1c8b104fa3affff363 P fbc183e336c74b8a8d68f4624d9b375b: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-2/wal/wals/ecd9c6c9a2b74a1c8b104fa3affff363
I20260925 15:20:41.595755 11312 ts_tablet_manager.cc:1939] T ecd9c6c9a2b74a1c8b104fa3affff363 P fbc183e336c74b8a8d68f4624d9b375b: Deleting consensus metadata
W20260925 15:20:41.596284 11144 connection.cc:441] server connection from 127.0.0.1:46910 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:46910 (request call id 1) could send its response
I20260925 15:20:41.601840 11268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:41.586652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50432 (local address 127.3.211.252:36373)
0925 15:20:41.586699 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:41.586702 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:41.592604 (+ 5902us) server_negotiation.cc:407] Connection header received
0925 15:20:41.592669 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:41.592670 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:41.592690 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:41.592757 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:41.594006 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.594769 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.596322 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.596670 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.596751 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:41.598493 (+ 1742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:41.598495 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:41.598496 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:41.598504 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:41.600374 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:41.600789 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:41.600790 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:41.600791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:41.600860 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:41.601107 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:41.601108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:41.601109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:41.601276 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:41.601387 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:41.601511 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 15:20:41.601606 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 15:20:41.605140 11271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:41.596869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52968 (local address 127.3.211.253:42795)
0925 15:20:41.596921 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:41.596923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:41.596932 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:20:41.597080 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:41.597082 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:41.597098 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:41.597128 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:41.597834 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.598330 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.600176 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.600537 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.600618 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:41.602101 (+ 1483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:41.602103 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:41.602103 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:41.602111 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:41.603755 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:41.604115 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:41.604116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:41.604116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:41.604178 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:41.604435 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:41.604436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:41.604437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:41.604580 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:41.604707 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:41.604948 (+ 241us) server_negotiation.cc:299] Negotiation successful
0925 15:20:41.605039 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 15:20:41.611578 10986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:41.597003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50446 (local address 127.3.211.252:36373)
0925 15:20:41.597047 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:41.597049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:41.604509 (+ 7460us) server_negotiation.cc:407] Connection header received
0925 15:20:41.604581 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:41.604582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:41.604593 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:41.604641 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:41.605278 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.605959 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.606922 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:41.607128 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:41.607214 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:41.608584 (+ 1370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:41.608586 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:41.608587 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:41.608594 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:41.610045 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:41.610436 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:41.610437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:41.610438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:41.610494 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:41.610875 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:41.610876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:41.610876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:41.611033 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:41.611125 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:41.611385 (+ 260us) server_negotiation.cc:299] Negotiation successful
0925 15:20:41.611469 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 15:20:42.580771 11320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:42.580461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38373 (local address 127.3.211.193:39773)
0925 15:20:42.580625 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:42.580704 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":62,"threads_started":1}
W20260925 15:20:42.581246 10983 heartbeater.cc:644] Failed to heartbeat to 127.3.211.254:38373 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.211.254:38373: connect: Connection refused (error 111)
I20260925 15:20:42.582232 11321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:42.581978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38373 (local address 127.3.211.194:55531)
0925 15:20:42.582102 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:42.582168 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":80,"threads_started":1}
W20260925 15:20:42.582532 11120 heartbeater.cc:644] Failed to heartbeat to 127.3.211.254:38373 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.211.254:38373: connect: Connection refused (error 111)
I20260925 15:20:42.582971 11322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:42.582607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38373 (local address 127.3.211.195:58069)
0925 15:20:42.582757 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:42.582917 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":62,"threads_started":1}
W20260925 15:20:42.583182 11258 heartbeater.cc:644] Failed to heartbeat to 127.3.211.254:38373 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.211.254:38373: connect: Connection refused (error 111)
I20260925 15:20:43.086516 11323 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b3860a4600f94b6fa48e2ac45ace80e8)
I20260925 15:20:43.086556 11324 raft_consensus.cc:482] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b3860a4600f94b6fa48e2ac45ace80e8)
I20260925 15:20:43.086702 11324 raft_consensus.cc:504] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38373 } } peers { permanent_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42795 } attrs { promote: false } } peers { permanent_uuid: "881c1ed0940a4a7483c36869cba574f8" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 36373 } attrs { promote: false } }
I20260925 15:20:43.086702 11323 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38373 } } peers { permanent_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42795 } attrs { promote: false } } peers { permanent_uuid: "881c1ed0940a4a7483c36869cba574f8" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 36373 } attrs { promote: false } }
I20260925 15:20:43.087019 11324 leader_election.cc:288] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b3860a4600f94b6fa48e2ac45ace80e8 (127.3.211.254:38373), 6a04d4b30a3b47d0a897ed7f3afa45a3 (127.3.211.253:42795)
I20260925 15:20:43.087262 11323 leader_election.cc:288] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b3860a4600f94b6fa48e2ac45ace80e8 (127.3.211.254:38373), 881c1ed0940a4a7483c36869cba574f8 (127.3.211.252:36373)
I20260925 15:20:43.087356 11325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:43.087174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38373 (local address 127.0.0.1:48472)
0925 15:20:43.087280 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:43.087326 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":57,"threads_started":1}
W20260925 15:20:43.087500 10789 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.211.254:38373: connect: Connection refused (error 111)
I20260925 15:20:43.087965 11329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:43.087770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38373 (local address 127.0.0.1:48480)
0925 15:20:43.087876 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:43.087929 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":48,"threads_started":1}
W20260925 15:20:43.088078 10603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.211.254:38373: connect: Connection refused (error 111)
I20260925 15:20:43.088481 11329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:43.088375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38373 (local address 127.0.0.1:48494)
0925 15:20:43.088427 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:43.088452 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41,"thread_start_us":66,"threads_started":1}
W20260925 15:20:43.088685 10603 leader_election.cc:334] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b3860a4600f94b6fa48e2ac45ace80e8 (127.3.211.254:38373): Network error: Client connection negotiation failed: client connection to 127.3.211.254:38373: connect: Connection refused (error 111)
I20260925 15:20:43.089494 11330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:43.087787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38373 (local address 127.0.0.1:48482)
0925 15:20:43.089415 (+ 1628us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:43.089455 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1605,"thread_start_us":86,"threads_started":1}
W20260925 15:20:43.089581 10789 leader_election.cc:334] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b3860a4600f94b6fa48e2ac45ace80e8 (127.3.211.254:38373): Network error: Client connection negotiation failed: client connection to 127.3.211.254:38373: connect: Connection refused (error 111)
I20260925 15:20:43.091415 11325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:43.087476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.253:42795 (local address 127.0.0.1:51316)
0925 15:20:43.087545 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:43.087579 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:43.087638 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:43.088098 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:43.088098 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:43.088101 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:20:43.088152 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:43.088154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:43.089379 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:43.089380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:43.090327 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:43.090340 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:43.091145 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:43.091147 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:43.091218 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:43.091224 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 15:20:43.091287 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 15:20:43.091331 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 15:20:43.091429 11331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:43.087834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51316 (local address 127.3.211.253:42795)
0925 15:20:43.087930 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:43.087933 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:43.087943 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:20:43.087972 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:43.087972 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:43.087983 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:43.088225 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:20:43.088265 (+ 40us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:43.089258 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:43.090432 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:43.091011 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:43.091096 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:43.091110 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:43.091308 (+ 198us) server_negotiation.cc:299] Negotiation successful
0925 15:20:43.091354 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":56,"threads_started":1}
I20260925 15:20:43.091536 11327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:43.087526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57410 (local address 127.3.211.252:36373)
0925 15:20:43.087940 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:43.087942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:43.087951 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:20:43.087972 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:43.087972 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:43.087983 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:43.088059 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:20:43.088779 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:43.089508 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:43.090651 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:43.091196 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:43.091263 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:43.091274 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:43.091416 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 15:20:43.091465 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":69,"threads_started":1}
I20260925 15:20:43.091539 11326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:43.087175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.252:36373 (local address 127.0.0.1:57410)
0925 15:20:43.087470 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:43.087483 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:43.087560 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:43.088256 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:43.088257 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:43.088259 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:20:43.088314 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:43.088316 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:43.089643 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:43.089644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:43.090498 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:43.090499 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:43.091283 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:43.091284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:43.091346 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:43.091350 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:20:43.091398 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 15:20:43.091446 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":54,"threads_started":1}
I20260925 15:20:43.091712 10627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "881c1ed0940a4a7483c36869cba574f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" is_pre_election: true
I20260925 15:20:43.091840 10627 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 881c1ed0940a4a7483c36869cba574f8 in term 1.
I20260925 15:20:43.091866 10813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "881c1ed0940a4a7483c36869cba574f8" is_pre_election: true
I20260925 15:20:43.091974 10813 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a04d4b30a3b47d0a897ed7f3afa45a3 in term 1.
I20260925 15:20:43.092032 10788 leader_election.cc:302] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6a04d4b30a3b47d0a897ed7f3afa45a3, 881c1ed0940a4a7483c36869cba574f8; no voters: b3860a4600f94b6fa48e2ac45ace80e8
I20260925 15:20:43.092147 11324 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 15:20:43.092172 11324 raft_consensus.cc:482] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b3860a4600f94b6fa48e2ac45ace80e8)
I20260925 15:20:43.092187 11324 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [term 1 FOLLOWER]: Advancing to term 2
I20260925 15:20:43.092259 10601 leader_election.cc:302] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6a04d4b30a3b47d0a897ed7f3afa45a3, 881c1ed0940a4a7483c36869cba574f8; no voters: b3860a4600f94b6fa48e2ac45ace80e8
I20260925 15:20:43.092399 11323 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 15:20:43.092434 11323 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b3860a4600f94b6fa48e2ac45ace80e8)
I20260925 15:20:43.092453 11323 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3 [term 1 FOLLOWER]: Advancing to term 2
I20260925 15:20:43.093114 11324 raft_consensus.cc:504] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38373 } } peers { permanent_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42795 } attrs { promote: false } } peers { permanent_uuid: "881c1ed0940a4a7483c36869cba574f8" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 36373 } attrs { promote: false } }
I20260925 15:20:43.093263 11323 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38373 } } peers { permanent_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42795 } attrs { promote: false } } peers { permanent_uuid: "881c1ed0940a4a7483c36869cba574f8" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 36373 } attrs { promote: false } }
I20260925 15:20:43.093397 11323 leader_election.cc:288] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3 [CANDIDATE]: Term 2 election: Requested vote from peers b3860a4600f94b6fa48e2ac45ace80e8 (127.3.211.254:38373), 881c1ed0940a4a7483c36869cba574f8 (127.3.211.252:36373)
I20260925 15:20:43.093510 11325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:43.093405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38373 (local address 127.0.0.1:48500)
0925 15:20:43.093446 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:43.093477 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260925 15:20:43.093510 10627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "881c1ed0940a4a7483c36869cba574f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3"
I20260925 15:20:43.093562 11326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:43.093459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38373 (local address 127.0.0.1:48502)
0925 15:20:43.093496 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:43.093522 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260925 15:20:43.093565 10813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "881c1ed0940a4a7483c36869cba574f8"
I20260925 15:20:43.093616 10627 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 881c1ed0940a4a7483c36869cba574f8 in current term 2: Already voted for candidate 6a04d4b30a3b47d0a897ed7f3afa45a3 in this term.
I20260925 15:20:43.093647 10813 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 6a04d4b30a3b47d0a897ed7f3afa45a3 in current term 2: Already voted for candidate 881c1ed0940a4a7483c36869cba574f8 in this term.
I20260925 15:20:43.093739 11324 leader_election.cc:288] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [CANDIDATE]: Term 2 election: Requested vote from peers b3860a4600f94b6fa48e2ac45ace80e8 (127.3.211.254:38373), 6a04d4b30a3b47d0a897ed7f3afa45a3 (127.3.211.253:42795)
I20260925 15:20:43.093904 11325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:43.093772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38373 (local address 127.0.0.1:48508)
0925 15:20:43.093843 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:43.093874 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 15:20:43.094002 11326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:43.093925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38373 (local address 127.0.0.1:48518)
0925 15:20:43.093962 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:43.093982 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260925 15:20:43.094007 10789 leader_election.cc:334] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b3860a4600f94b6fa48e2ac45ace80e8 (127.3.211.254:38373): Network error: Client connection negotiation failed: client connection to 127.3.211.254:38373: connect: Connection refused (error 111)
I20260925 15:20:43.094056 10789 leader_election.cc:302] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 881c1ed0940a4a7483c36869cba574f8; no voters: 6a04d4b30a3b47d0a897ed7f3afa45a3, b3860a4600f94b6fa48e2ac45ace80e8
W20260925 15:20:43.094077 10603 leader_election.cc:334] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b3860a4600f94b6fa48e2ac45ace80e8 (127.3.211.254:38373): Network error: Client connection negotiation failed: client connection to 127.3.211.254:38373: connect: Connection refused (error 111)
I20260925 15:20:43.094110 10603 leader_election.cc:302] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6a04d4b30a3b47d0a897ed7f3afa45a3; no voters: 881c1ed0940a4a7483c36869cba574f8, b3860a4600f94b6fa48e2ac45ace80e8
I20260925 15:20:43.094180 11324 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 15:20:43.094202 11323 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 15:20:43.582022 11334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:43.581736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38373 (local address 127.3.211.193:44319)
0925 15:20:43.581904 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:43.581959 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":87,"threads_started":1}
I20260925 15:20:43.583110 11335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:43.582878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38373 (local address 127.3.211.194:48187)
0925 15:20:43.583000 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:43.583042 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":51,"threads_started":1}
I20260925 15:20:43.583715 11336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:43.583505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38373 (local address 127.3.211.195:43257)
0925 15:20:43.583638 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:43.583672 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":68,"threads_started":1}
I20260925 15:20:44.582859 11337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:44.582621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38373 (local address 127.3.211.193:36389)
0925 15:20:44.582759 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:44.582803 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":90,"threads_started":1}
I20260925 15:20:44.583961 11338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:44.583757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38373 (local address 127.3.211.194:46685)
0925 15:20:44.583878 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:44.583913 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":53,"threads_started":1}
I20260925 15:20:44.584405 11339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:44.584203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38373 (local address 127.3.211.195:43569)
0925 15:20:44.584294 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:44.584359 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76,"thread_start_us":48,"threads_started":1}
I20260925 15:20:44.913928 11340 raft_consensus.cc:482] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:20:44.914040 11340 raft_consensus.cc:504] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38373 } } peers { permanent_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42795 } attrs { promote: false } } peers { permanent_uuid: "881c1ed0940a4a7483c36869cba574f8" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 36373 } attrs { promote: false } }
I20260925 15:20:44.914294 11340 leader_election.cc:288] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b3860a4600f94b6fa48e2ac45ace80e8 (127.3.211.254:38373), 6a04d4b30a3b47d0a897ed7f3afa45a3 (127.3.211.253:42795)
I20260925 15:20:44.914577 10627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "881c1ed0940a4a7483c36869cba574f8" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" is_pre_election: true
I20260925 15:20:44.914646 11341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:44.914437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38373 (local address 127.0.0.1:48786)
0925 15:20:44.914564 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:44.914609 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":68,"threads_started":1}
I20260925 15:20:44.914729 10627 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 881c1ed0940a4a7483c36869cba574f8 in term 2.
I20260925 15:20:44.914919 10788 leader_election.cc:302] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a04d4b30a3b47d0a897ed7f3afa45a3, 881c1ed0940a4a7483c36869cba574f8; no voters:
I20260925 15:20:44.915051 11340 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 15:20:44.915099 11341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:44.915010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38373 (local address 127.0.0.1:48796)
0925 15:20:44.915043 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:44.915072 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260925 15:20:44.915102 11340 raft_consensus.cc:482] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:20:44.915169 11340 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [term 2 FOLLOWER]: Advancing to term 3
W20260925 15:20:44.915174 10789 leader_election.cc:334] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b3860a4600f94b6fa48e2ac45ace80e8 (127.3.211.254:38373): Network error: Client connection negotiation failed: client connection to 127.3.211.254:38373: connect: Connection refused (error 111)
I20260925 15:20:44.915943 11340 raft_consensus.cc:504] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38373 } } peers { permanent_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42795 } attrs { promote: false } } peers { permanent_uuid: "881c1ed0940a4a7483c36869cba574f8" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 36373 } attrs { promote: false } }
I20260925 15:20:44.916100 11340 leader_election.cc:288] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [CANDIDATE]: Term 3 election: Requested vote from peers b3860a4600f94b6fa48e2ac45ace80e8 (127.3.211.254:38373), 6a04d4b30a3b47d0a897ed7f3afa45a3 (127.3.211.253:42795)
I20260925 15:20:44.916292 10627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "881c1ed0940a4a7483c36869cba574f8" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3"
I20260925 15:20:44.916373 11341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:44.916218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38373 (local address 127.0.0.1:48800)
0925 15:20:44.916295 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:44.916324 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 15:20:44.916421 10627 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3 [term 2 FOLLOWER]: Advancing to term 3
I20260925 15:20:44.916754 11341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:44.916605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38373 (local address 127.0.0.1:48816)
0925 15:20:44.916675 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:44.916729 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 15:20:44.916843 10789 leader_election.cc:334] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer b3860a4600f94b6fa48e2ac45ace80e8 (127.3.211.254:38373): Network error: Client connection negotiation failed: client connection to 127.3.211.254:38373: connect: Connection refused (error 111)
I20260925 15:20:44.917171 10627 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 881c1ed0940a4a7483c36869cba574f8 in term 3.
I20260925 15:20:44.917336 10788 leader_election.cc:302] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6a04d4b30a3b47d0a897ed7f3afa45a3, 881c1ed0940a4a7483c36869cba574f8; no voters: b3860a4600f94b6fa48e2ac45ace80e8
I20260925 15:20:44.917452 11340 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 15:20:44.917567 11340 raft_consensus.cc:686] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [term 3 LEADER]: Becoming Leader. State: Replica: 881c1ed0940a4a7483c36869cba574f8, State: Running, Role: LEADER
I20260925 15:20:44.917640 11340 consensus_queue.cc:237] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38373 } } peers { permanent_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42795 } attrs { promote: false } } peers { permanent_uuid: "881c1ed0940a4a7483c36869cba574f8" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 36373 } attrs { promote: false } }
I20260925 15:20:44.918037 11343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 881c1ed0940a4a7483c36869cba574f8. Latest consensus state: current_term: 3 leader_uuid: "881c1ed0940a4a7483c36869cba574f8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38373 } } peers { permanent_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42795 } attrs { promote: false } } peers { permanent_uuid: "881c1ed0940a4a7483c36869cba574f8" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 36373 } attrs { promote: false } } }
I20260925 15:20:44.918123 11343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [sys.catalog]: This master's current role is: LEADER
I20260925 15:20:44.918275 11345 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:20:44.918758 11345 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:20:44.918824 11345 catalog_manager.cc:1259] Loaded cluster ID: f33e811646eb40de953a49f346770393
I20260925 15:20:44.918883 11345 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:20:44.919081 11345 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:20:44.919174 11345 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8: Loaded TSK: 0
I20260925 15:20:44.919348 11345 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:20:44.944833 10803 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50446:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:20:44.946455 10803 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:20:44.947564 11341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:44.947402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38373 (local address 127.0.0.1:48834)
0925 15:20:44.947485 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:44.947526 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 15:20:44.947621 10627 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3 [term 3 FOLLOWER]: Refusing update from remote peer 881c1ed0940a4a7483c36869cba574f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260925 15:20:44.947902 11340 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42795 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260925 15:20:44.948022 11341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:44.947916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38373 (local address 127.0.0.1:48842)
0925 15:20:44.947968 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:44.947992 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 15:20:44.948184 10789 consensus_peers.cc:487] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 -> Peer b3860a4600f94b6fa48e2ac45ace80e8 (127.3.211.254:38373): Couldn't send request to peer b3860a4600f94b6fa48e2ac45ace80e8. Status: Network error: Client connection negotiation failed: client connection to 127.3.211.254:38373: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 15:20:44.948582 11347 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 881c1ed0940a4a7483c36869cba574f8. Latest consensus state: current_term: 3 leader_uuid: "881c1ed0940a4a7483c36869cba574f8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38373 } } peers { permanent_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42795 } attrs { promote: false } } peers { permanent_uuid: "881c1ed0940a4a7483c36869cba574f8" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 36373 } attrs { promote: false } } }
I20260925 15:20:44.948685 11347 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:20:44.949115 11346 mvcc.cc:205] Tried to move back new op lower bound from 7333272145702944768 to 7333272145583435776. Current Snapshot: MvccSnapshot[applied={T|T < 7333272145702944768}]
I20260925 15:20:44.949247 11340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "881c1ed0940a4a7483c36869cba574f8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38373 } } peers { permanent_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42795 } attrs { promote: false } } peers { permanent_uuid: "881c1ed0940a4a7483c36869cba574f8" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 36373 } attrs { promote: false } } }
I20260925 15:20:44.949344 11340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8 [sys.catalog]: This master's current role is: LEADER
I20260925 15:20:44.949502 11347 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "881c1ed0940a4a7483c36869cba574f8" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b3860a4600f94b6fa48e2ac45ace80e8" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38373 } } peers { permanent_uuid: "6a04d4b30a3b47d0a897ed7f3afa45a3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42795 } attrs { promote: false } } peers { permanent_uuid: "881c1ed0940a4a7483c36869cba574f8" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 36373 } attrs { promote: false } } }
I20260925 15:20:44.949781 11347 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a04d4b30a3b47d0a897ed7f3afa45a3 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:20:44.955287 11352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:44.951524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51352 (local address 127.3.211.193:32847)
0925 15:20:44.951790 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:44.951792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:44.951799 (+ 7us) server_negotiation.cc:407] Connection header received
0925 15:20:44.951826 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:44.951826 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:44.951836 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:44.951940 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:20:44.952063 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:44.952710 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:44.954182 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:44.954868 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:44.954965 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:44.954978 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:44.955140 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 15:20:44.955194 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":72,"threads_started":1}
I20260925 15:20:44.955303 11341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:44.951460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:32847 (local address 127.0.0.1:51352)
0925 15:20:44.951511 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:44.951524 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:44.951575 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:44.951941 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:44.951942 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:44.951944 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:20:44.952000 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:44.952002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:44.952797 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:44.952798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:44.954069 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:44.954072 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:44.954981 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:44.954982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:44.955047 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:44.955053 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 15:20:44.955127 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 15:20:44.955192 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":1}
I20260925 15:20:44.955382 11355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:44.951648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34462 (local address 127.3.211.195:34379)
0925 15:20:44.951743 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:44.951745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:44.952013 (+ 268us) server_negotiation.cc:407] Connection header received
0925 15:20:44.952044 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:44.952045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:44.952057 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:44.952138 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:20:44.952225 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:44.953049 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:44.954319 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:44.954902 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:44.954965 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:44.954978 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:44.955257 (+ 279us) server_negotiation.cc:299] Negotiation successful
0925 15:20:44.955308 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":48,"threads_started":1}
I20260925 15:20:44.955480 11354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:44.951583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57070 (local address 127.3.211.194:44803)
0925 15:20:44.951713 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:44.951716 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:44.952133 (+ 417us) server_negotiation.cc:407] Connection header received
0925 15:20:44.952162 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:44.952162 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:44.952173 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:44.952242 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:20:44.952430 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:44.953145 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:44.954511 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:44.954954 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:44.955033 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:44.955046 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:44.955338 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 15:20:44.955390 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":76,"threads_started":1}
I20260925 15:20:44.955541 11356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:44.951510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.195:34379 (local address 127.0.0.1:34462)
0925 15:20:44.951835 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:44.951846 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:44.951902 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:44.952139 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:44.952140 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:44.952142 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:20:44.952171 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:44.952172 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:44.953151 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:44.953152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:44.954113 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:44.954115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:44.955015 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:44.955016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:44.955085 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:44.955111 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 15:20:44.955162 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 15:20:44.955210 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":42,"thread_start_us":90,"threads_started":1}
I20260925 15:20:44.955308 11353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:44.951461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.194:44803 (local address 127.0.0.1:57070)
0925 15:20:44.952013 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:44.952024 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:44.952076 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:44.952286 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:44.952287 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:44.952289 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:20:44.952353 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:44.952354 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:44.953255 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:44.953257 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:44.954411 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:44.954414 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:44.955077 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:44.955078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:44.955138 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:44.955141 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:20:44.955190 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 15:20:44.955227 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"mutex_wait_us":164,"thread_start_us":79,"threads_started":1}
I20260925 15:20:44.955833 11053 tablet_service.cc:1467] Processing CreateTablet for tablet 96d41cf8068047a9a79a78c49fe9ffbd (DEFAULT_TABLE table=test-table [id=be228236955f4757be3b206f4800f523]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:20:44.955984 11053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96d41cf8068047a9a79a78c49fe9ffbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:44.956385 11191 tablet_service.cc:1467] Processing CreateTablet for tablet 96d41cf8068047a9a79a78c49fe9ffbd (DEFAULT_TABLE table=test-table [id=be228236955f4757be3b206f4800f523]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:20:44.956529 11191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96d41cf8068047a9a79a78c49fe9ffbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:44.956506 10916 tablet_service.cc:1467] Processing CreateTablet for tablet 96d41cf8068047a9a79a78c49fe9ffbd (DEFAULT_TABLE table=test-table [id=be228236955f4757be3b206f4800f523]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:20:44.956605 10916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96d41cf8068047a9a79a78c49fe9ffbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:44.957391 11357 tablet_bootstrap.cc:492] T 96d41cf8068047a9a79a78c49fe9ffbd P 7921060cf7c04ccf844a869c26b58c4a: Bootstrap starting.
I20260925 15:20:44.957734 11357 tablet_bootstrap.cc:654] T 96d41cf8068047a9a79a78c49fe9ffbd P 7921060cf7c04ccf844a869c26b58c4a: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:44.958194 11360 tablet_bootstrap.cc:492] T 96d41cf8068047a9a79a78c49fe9ffbd P bf6c0d3ea9e342d6960a6f8a5d1e7e26: Bootstrap starting.
I20260925 15:20:44.958338 11357 tablet_bootstrap.cc:492] T 96d41cf8068047a9a79a78c49fe9ffbd P 7921060cf7c04ccf844a869c26b58c4a: No bootstrap required, opened a new log
I20260925 15:20:44.958381 11357 ts_tablet_manager.cc:1397] T 96d41cf8068047a9a79a78c49fe9ffbd P 7921060cf7c04ccf844a869c26b58c4a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:44.958477 11357 raft_consensus.cc:348] T 96d41cf8068047a9a79a78c49fe9ffbd P 7921060cf7c04ccf844a869c26b58c4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7921060cf7c04ccf844a869c26b58c4a" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44803 } } peers { permanent_uuid: "bf6c0d3ea9e342d6960a6f8a5d1e7e26" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 32847 } } peers { permanent_uuid: "fbc183e336c74b8a8d68f4624d9b375b" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 34379 } }
I20260925 15:20:44.958534 11357 raft_consensus.cc:374] T 96d41cf8068047a9a79a78c49fe9ffbd P 7921060cf7c04ccf844a869c26b58c4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:44.958546 11357 raft_consensus.cc:729] T 96d41cf8068047a9a79a78c49fe9ffbd P 7921060cf7c04ccf844a869c26b58c4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7921060cf7c04ccf844a869c26b58c4a, State: Initialized, Role: FOLLOWER
I20260925 15:20:44.958616 11360 tablet_bootstrap.cc:654] T 96d41cf8068047a9a79a78c49fe9ffbd P bf6c0d3ea9e342d6960a6f8a5d1e7e26: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:44.958607 11357 consensus_queue.cc:260] T 96d41cf8068047a9a79a78c49fe9ffbd P 7921060cf7c04ccf844a869c26b58c4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7921060cf7c04ccf844a869c26b58c4a" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44803 } } peers { permanent_uuid: "bf6c0d3ea9e342d6960a6f8a5d1e7e26" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 32847 } } peers { permanent_uuid: "fbc183e336c74b8a8d68f4624d9b375b" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 34379 } }
I20260925 15:20:44.958827 11357 ts_tablet_manager.cc:1428] T 96d41cf8068047a9a79a78c49fe9ffbd P 7921060cf7c04ccf844a869c26b58c4a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 15:20:44.959128 11338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:44.959030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38373 (local address 127.3.211.194:44569)
0925 15:20:44.959066 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:44.959100 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260925 15:20:44.959204 11360 tablet_bootstrap.cc:492] T 96d41cf8068047a9a79a78c49fe9ffbd P bf6c0d3ea9e342d6960a6f8a5d1e7e26: No bootstrap required, opened a new log
I20260925 15:20:44.959254 11360 ts_tablet_manager.cc:1397] T 96d41cf8068047a9a79a78c49fe9ffbd P bf6c0d3ea9e342d6960a6f8a5d1e7e26: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:44.959380 11360 raft_consensus.cc:348] T 96d41cf8068047a9a79a78c49fe9ffbd P bf6c0d3ea9e342d6960a6f8a5d1e7e26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7921060cf7c04ccf844a869c26b58c4a" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44803 } } peers { permanent_uuid: "bf6c0d3ea9e342d6960a6f8a5d1e7e26" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 32847 } } peers { permanent_uuid: "fbc183e336c74b8a8d68f4624d9b375b" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 34379 } }
I20260925 15:20:44.959451 11360 raft_consensus.cc:374] T 96d41cf8068047a9a79a78c49fe9ffbd P bf6c0d3ea9e342d6960a6f8a5d1e7e26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:44.959470 11360 raft_consensus.cc:729] T 96d41cf8068047a9a79a78c49fe9ffbd P bf6c0d3ea9e342d6960a6f8a5d1e7e26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf6c0d3ea9e342d6960a6f8a5d1e7e26, State: Initialized, Role: FOLLOWER
I20260925 15:20:44.959471 11119 heartbeater.cc:503] Master 127.3.211.252:36373 was elected leader, sending a full tablet report...
I20260925 15:20:44.959513 11360 consensus_queue.cc:260] T 96d41cf8068047a9a79a78c49fe9ffbd P bf6c0d3ea9e342d6960a6f8a5d1e7e26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7921060cf7c04ccf844a869c26b58c4a" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44803 } } peers { permanent_uuid: "bf6c0d3ea9e342d6960a6f8a5d1e7e26" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 32847 } } peers { permanent_uuid: "fbc183e336c74b8a8d68f4624d9b375b" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 34379 } }
I20260925 15:20:44.959682 11360 ts_tablet_manager.cc:1428] T 96d41cf8068047a9a79a78c49fe9ffbd P bf6c0d3ea9e342d6960a6f8a5d1e7e26: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:44.959707 11338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:44.959624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38373 (local address 127.3.211.194:49585)
0925 15:20:44.959656 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:44.959681 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260925 15:20:44.960009 11337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:44.959924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38373 (local address 127.3.211.193:55529)
0925 15:20:44.959963 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:44.959985 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260925 15:20:44.960316 10982 heartbeater.cc:503] Master 127.3.211.252:36373 was elected leader, sending a full tablet report...
I20260925 15:20:44.960464 11337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:44.960380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38373 (local address 127.3.211.193:52879)
0925 15:20:44.960415 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:44.960438 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260925 15:20:44.960668 11359 tablet_bootstrap.cc:492] T 96d41cf8068047a9a79a78c49fe9ffbd P fbc183e336c74b8a8d68f4624d9b375b: Bootstrap starting.
I20260925 15:20:44.961140 11359 tablet_bootstrap.cc:654] T 96d41cf8068047a9a79a78c49fe9ffbd P fbc183e336c74b8a8d68f4624d9b375b: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:44.961795 11359 tablet_bootstrap.cc:492] T 96d41cf8068047a9a79a78c49fe9ffbd P fbc183e336c74b8a8d68f4624d9b375b: No bootstrap required, opened a new log
I20260925 15:20:44.961841 11359 ts_tablet_manager.cc:1397] T 96d41cf8068047a9a79a78c49fe9ffbd P fbc183e336c74b8a8d68f4624d9b375b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:44.961987 11359 raft_consensus.cc:348] T 96d41cf8068047a9a79a78c49fe9ffbd P fbc183e336c74b8a8d68f4624d9b375b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7921060cf7c04ccf844a869c26b58c4a" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44803 } } peers { permanent_uuid: "bf6c0d3ea9e342d6960a6f8a5d1e7e26" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 32847 } } peers { permanent_uuid: "fbc183e336c74b8a8d68f4624d9b375b" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 34379 } }
I20260925 15:20:44.962045 11359 raft_consensus.cc:374] T 96d41cf8068047a9a79a78c49fe9ffbd P fbc183e336c74b8a8d68f4624d9b375b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:44.962088 11359 raft_consensus.cc:729] T 96d41cf8068047a9a79a78c49fe9ffbd P fbc183e336c74b8a8d68f4624d9b375b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbc183e336c74b8a8d68f4624d9b375b, State: Initialized, Role: FOLLOWER
I20260925 15:20:44.962168 11359 consensus_queue.cc:260] T 96d41cf8068047a9a79a78c49fe9ffbd P fbc183e336c74b8a8d68f4624d9b375b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7921060cf7c04ccf844a869c26b58c4a" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44803 } } peers { permanent_uuid: "bf6c0d3ea9e342d6960a6f8a5d1e7e26" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 32847 } } peers { permanent_uuid: "fbc183e336c74b8a8d68f4624d9b375b" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 34379 } }
I20260925 15:20:44.962364 11359 ts_tablet_manager.cc:1428] T 96d41cf8068047a9a79a78c49fe9ffbd P fbc183e336c74b8a8d68f4624d9b375b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 15:20:44.962666 11339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:44.962573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38373 (local address 127.3.211.195:41749)
0925 15:20:44.962608 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:44.962643 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260925 15:20:44.963017 11257 heartbeater.cc:503] Master 127.3.211.252:36373 was elected leader, sending a full tablet report...
I20260925 15:20:44.963023 11339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:44.962910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38373 (local address 127.3.211.195:36293)
0925 15:20:44.962965 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:44.962994 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 15:20:44.975198 11361 raft_consensus.cc:482] T 96d41cf8068047a9a79a78c49fe9ffbd P 7921060cf7c04ccf844a869c26b58c4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:20:44.975308 11361 raft_consensus.cc:504] T 96d41cf8068047a9a79a78c49fe9ffbd P 7921060cf7c04ccf844a869c26b58c4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7921060cf7c04ccf844a869c26b58c4a" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44803 } } peers { permanent_uuid: "bf6c0d3ea9e342d6960a6f8a5d1e7e26" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 32847 } } peers { permanent_uuid: "fbc183e336c74b8a8d68f4624d9b375b" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 34379 } }
I20260925 15:20:44.975741 11361 leader_election.cc:288] T 96d41cf8068047a9a79a78c49fe9ffbd P 7921060cf7c04ccf844a869c26b58c4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf6c0d3ea9e342d6960a6f8a5d1e7e26 (127.3.211.193:32847), fbc183e336c74b8a8d68f4624d9b375b (127.3.211.195:34379)
I20260925 15:20:44.978843 11338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:44.975952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.195:34379 (local address 127.3.211.194:35447)
0925 15:20:44.976006 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:44.976019 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:44.976083 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:44.976290 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:44.976291 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:44.976295 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:20:44.976406 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:44.976408 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:44.977094 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:44.977095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:44.977945 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:44.977948 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:44.978563 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:44.978564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:44.978635 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:44.978640 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:20:44.978697 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 15:20:44.978747 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 15:20:44.978843 11355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:44.976015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:35447 (local address 127.3.211.195:34379)
0925 15:20:44.976063 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:44.976065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:44.976076 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:20:44.976185 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:44.976186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:44.976198 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:44.976258 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:20:44.976478 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:44.976995 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:44.978050 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:44.978477 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:44.978542 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:44.978552 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:44.978713 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 15:20:44.978759 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 15:20:44.978935 11352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:44.976044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:41723 (local address 127.3.211.193:32847)
0925 15:20:44.976104 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:44.976106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:44.976300 (+ 194us) server_negotiation.cc:407] Connection header received
0925 15:20:44.976360 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:44.976361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:44.976373 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:44.976441 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:20:44.976619 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:44.977139 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:44.978160 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:44.978538 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:44.978605 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:44.978614 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:44.978821 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 15:20:44.978864 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 15:20:44.979007 11366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:44.975965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:32847 (local address 127.3.211.194:41723)
0925 15:20:44.976222 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:44.976233 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:44.976287 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:44.976465 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:44.976466 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:44.976468 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:20:44.976528 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:44.976530 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:44.977249 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:44.977250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:44.978061 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:44.978064 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:44.978649 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:44.978650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:44.978737 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:44.978742 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:20:44.978807 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 15:20:44.978845 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":48,"thread_start_us":122,"threads_started":1}
I20260925 15:20:44.979135 11211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96d41cf8068047a9a79a78c49fe9ffbd" candidate_uuid: "7921060cf7c04ccf844a869c26b58c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbc183e336c74b8a8d68f4624d9b375b" is_pre_election: true
I20260925 15:20:44.979223 11211 raft_consensus.cc:2454] T 96d41cf8068047a9a79a78c49fe9ffbd P fbc183e336c74b8a8d68f4624d9b375b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7921060cf7c04ccf844a869c26b58c4a in term 0.
I20260925 15:20:44.979350 10936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96d41cf8068047a9a79a78c49fe9ffbd" candidate_uuid: "7921060cf7c04ccf844a869c26b58c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf6c0d3ea9e342d6960a6f8a5d1e7e26" is_pre_election: true
I20260925 15:20:44.979414 11009 leader_election.cc:302] T 96d41cf8068047a9a79a78c49fe9ffbd P 7921060cf7c04ccf844a869c26b58c4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7921060cf7c04ccf844a869c26b58c4a, fbc183e336c74b8a8d68f4624d9b375b; no voters:
I20260925 15:20:44.979511 10936 raft_consensus.cc:2454] T 96d41cf8068047a9a79a78c49fe9ffbd P bf6c0d3ea9e342d6960a6f8a5d1e7e26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7921060cf7c04ccf844a869c26b58c4a in term 0.
I20260925 15:20:44.979565 11361 raft_consensus.cc:2781] T 96d41cf8068047a9a79a78c49fe9ffbd P 7921060cf7c04ccf844a869c26b58c4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:20:44.979610 11361 raft_consensus.cc:482] T 96d41cf8068047a9a79a78c49fe9ffbd P 7921060cf7c04ccf844a869c26b58c4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:20:44.979657 11361 raft_consensus.cc:3037] T 96d41cf8068047a9a79a78c49fe9ffbd P 7921060cf7c04ccf844a869c26b58c4a [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:44.980602 11361 raft_consensus.cc:504] T 96d41cf8068047a9a79a78c49fe9ffbd P 7921060cf7c04ccf844a869c26b58c4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7921060cf7c04ccf844a869c26b58c4a" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44803 } } peers { permanent_uuid: "bf6c0d3ea9e342d6960a6f8a5d1e7e26" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 32847 } } peers { permanent_uuid: "fbc183e336c74b8a8d68f4624d9b375b" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 34379 } }
I20260925 15:20:44.980756 11361 leader_election.cc:288] T 96d41cf8068047a9a79a78c49fe9ffbd P 7921060cf7c04ccf844a869c26b58c4a [CANDIDATE]: Term 1 election: Requested vote from peers bf6c0d3ea9e342d6960a6f8a5d1e7e26 (127.3.211.193:32847), fbc183e336c74b8a8d68f4624d9b375b (127.3.211.195:34379)
I20260925 15:20:44.980926 11211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96d41cf8068047a9a79a78c49fe9ffbd" candidate_uuid: "7921060cf7c04ccf844a869c26b58c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbc183e336c74b8a8d68f4624d9b375b"
I20260925 15:20:44.980944 10936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96d41cf8068047a9a79a78c49fe9ffbd" candidate_uuid: "7921060cf7c04ccf844a869c26b58c4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf6c0d3ea9e342d6960a6f8a5d1e7e26"
I20260925 15:20:44.981019 10936 raft_consensus.cc:3037] T 96d41cf8068047a9a79a78c49fe9ffbd P bf6c0d3ea9e342d6960a6f8a5d1e7e26 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:44.981022 11211 raft_consensus.cc:3037] T 96d41cf8068047a9a79a78c49fe9ffbd P fbc183e336c74b8a8d68f4624d9b375b [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:44.981617 11211 raft_consensus.cc:2454] T 96d41cf8068047a9a79a78c49fe9ffbd P fbc183e336c74b8a8d68f4624d9b375b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7921060cf7c04ccf844a869c26b58c4a in term 1.
I20260925 15:20:44.981752 10936 raft_consensus.cc:2454] T 96d41cf8068047a9a79a78c49fe9ffbd P bf6c0d3ea9e342d6960a6f8a5d1e7e26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7921060cf7c04ccf844a869c26b58c4a in term 1.
I20260925 15:20:44.981845 11009 leader_election.cc:302] T 96d41cf8068047a9a79a78c49fe9ffbd P 7921060cf7c04ccf844a869c26b58c4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7921060cf7c04ccf844a869c26b58c4a, fbc183e336c74b8a8d68f4624d9b375b; no voters:
I20260925 15:20:44.981935 11361 raft_consensus.cc:2781] T 96d41cf8068047a9a79a78c49fe9ffbd P 7921060cf7c04ccf844a869c26b58c4a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:20:44.982092 11361 raft_consensus.cc:686] T 96d41cf8068047a9a79a78c49fe9ffbd P 7921060cf7c04ccf844a869c26b58c4a [term 1 LEADER]: Becoming Leader. State: Replica: 7921060cf7c04ccf844a869c26b58c4a, State: Running, Role: LEADER
I20260925 15:20:44.982175 11361 consensus_queue.cc:237] T 96d41cf8068047a9a79a78c49fe9ffbd P 7921060cf7c04ccf844a869c26b58c4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7921060cf7c04ccf844a869c26b58c4a" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44803 } } peers { permanent_uuid: "bf6c0d3ea9e342d6960a6f8a5d1e7e26" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 32847 } } peers { permanent_uuid: "fbc183e336c74b8a8d68f4624d9b375b" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 34379 } }
I20260925 15:20:44.982424 11366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:44.982328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38373 (local address 127.3.211.194:41687)
0925 15:20:44.982363 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:44.982393 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260925 15:20:44.982973 10803 catalog_manager.cc:5495] T 96d41cf8068047a9a79a78c49fe9ffbd P 7921060cf7c04ccf844a869c26b58c4a reported cstate change: term changed from 0 to 1, leader changed from <none> to 7921060cf7c04ccf844a869c26b58c4a (127.3.211.194). New cstate: current_term: 1 leader_uuid: "7921060cf7c04ccf844a869c26b58c4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7921060cf7c04ccf844a869c26b58c4a" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf6c0d3ea9e342d6960a6f8a5d1e7e26" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 32847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbc183e336c74b8a8d68f4624d9b375b" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 34379 } health_report { overall_health: UNKNOWN } } }
I20260925 15:20:44.992949 11354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:44.990258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57086 (local address 127.3.211.194:44803)
0925 15:20:44.990390 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:44.990392 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:44.990401 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:20:44.990434 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:44.990435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:44.990453 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:44.990523 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:20:44.990696 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:44.991173 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:44.991978 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:44.992197 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:44.992263 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:44.992500 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:20:44.992606 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:20:44.992671 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:44.992844 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 15:20:44.992874 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260925 15:20:44.994813 11211 raft_consensus.cc:1261] T 96d41cf8068047a9a79a78c49fe9ffbd P fbc183e336c74b8a8d68f4624d9b375b [term 1 FOLLOWER]: Refusing update from remote peer 7921060cf7c04ccf844a869c26b58c4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:20:44.994902 10936 raft_consensus.cc:1261] T 96d41cf8068047a9a79a78c49fe9ffbd P bf6c0d3ea9e342d6960a6f8a5d1e7e26 [term 1 FOLLOWER]: Refusing update from remote peer 7921060cf7c04ccf844a869c26b58c4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:20:44.995129 11361 consensus_queue.cc:1035] T 96d41cf8068047a9a79a78c49fe9ffbd P 7921060cf7c04ccf844a869c26b58c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbc183e336c74b8a8d68f4624d9b375b" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 34379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:20:44.995338 11367 consensus_queue.cc:1035] T 96d41cf8068047a9a79a78c49fe9ffbd P 7921060cf7c04ccf844a869c26b58c4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf6c0d3ea9e342d6960a6f8a5d1e7e26" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 32847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:20:44.996272 11337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:44.996145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38373 (local address 127.3.211.193:42139)
0925 15:20:44.996193 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:44.996227 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 15:20:44.996699 11339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:44.996465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38373 (local address 127.3.211.195:47665)
0925 15:20:44.996609 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:44.996661 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260925 15:20:44.997895 11366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:44.997720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38373 (local address 127.3.211.194:59783)
0925 15:20:44.997831 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:44.997864 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260925 15:20:44.997987 11339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:44.997814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38373 (local address 127.3.211.195:48585)
0925 15:20:44.997925 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:44.997956 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260925 15:20:44.998337 11366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:44.998240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38373 (local address 127.3.211.194:53459)
0925 15:20:44.998274 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:44.998309 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260925 15:20:44.998876 11337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:44.998756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38373 (local address 127.3.211.193:45833)
0925 15:20:44.998823 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:44.998849 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:38373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 15:20:45.001060 10803 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:20:45.001187 10803 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:20:45.002530 10803 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 881c1ed0940a4a7483c36869cba574f8: Sending DeleteTablet for 3 replicas of tablet 96d41cf8068047a9a79a78c49fe9ffbd
I20260925 15:20:45.002887 11191 tablet_service.cc:1514] Processing DeleteTablet for tablet 96d41cf8068047a9a79a78c49fe9ffbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:20:44 UTC) from {username='oryx', principal='oryx/127.3.211.252@KRBTEST.COM'} at 127.0.0.1:34462
I20260925 15:20:45.002938 11053 tablet_service.cc:1514] Processing DeleteTablet for tablet 96d41cf8068047a9a79a78c49fe9ffbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:20:44 UTC) from {username='oryx', principal='oryx/127.3.211.252@KRBTEST.COM'} at 127.0.0.1:57070
I20260925 15:20:45.002947 10916 tablet_service.cc:1514] Processing DeleteTablet for tablet 96d41cf8068047a9a79a78c49fe9ffbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:20:44 UTC) from {username='oryx', principal='oryx/127.3.211.252@KRBTEST.COM'} at 127.0.0.1:51352
I20260925 15:20:45.003131 11381 tablet_replica.cc:330] T 96d41cf8068047a9a79a78c49fe9ffbd P fbc183e336c74b8a8d68f4624d9b375b: stopping tablet replica
I20260925 15:20:45.003211 11381 raft_consensus.cc:2229] T 96d41cf8068047a9a79a78c49fe9ffbd P fbc183e336c74b8a8d68f4624d9b375b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:20:45.003293 11381 raft_consensus.cc:2258] T 96d41cf8068047a9a79a78c49fe9ffbd P fbc183e336c74b8a8d68f4624d9b375b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:20:45.003548 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 10764
I20260925 15:20:45.003597 11381 ts_tablet_manager.cc:1905] T 96d41cf8068047a9a79a78c49fe9ffbd P fbc183e336c74b8a8d68f4624d9b375b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:20:45.005725 11381 ts_tablet_manager.cc:1918] T 96d41cf8068047a9a79a78c49fe9ffbd P fbc183e336c74b8a8d68f4624d9b375b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:20:45.005776 11381 log.cc:1196] T 96d41cf8068047a9a79a78c49fe9ffbd P fbc183e336c74b8a8d68f4624d9b375b: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-2/wal/wals/96d41cf8068047a9a79a78c49fe9ffbd
I20260925 15:20:45.005965 11381 ts_tablet_manager.cc:1939] T 96d41cf8068047a9a79a78c49fe9ffbd P fbc183e336c74b8a8d68f4624d9b375b: Deleting consensus metadata
I20260925 15:20:45.006542 10790 catalog_manager.cc:4841] TS fbc183e336c74b8a8d68f4624d9b375b (127.3.211.195:34379): tablet 96d41cf8068047a9a79a78c49fe9ffbd (table test-table [id=be228236955f4757be3b206f4800f523]) successfully deleted
I20260925 15:20:45.008042 11382 tablet_replica.cc:330] T 96d41cf8068047a9a79a78c49fe9ffbd P 7921060cf7c04ccf844a869c26b58c4a: stopping tablet replica
I20260925 15:20:45.008145 11382 raft_consensus.cc:2229] T 96d41cf8068047a9a79a78c49fe9ffbd P 7921060cf7c04ccf844a869c26b58c4a [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:20:45.008256 11382 raft_consensus.cc:2258] T 96d41cf8068047a9a79a78c49fe9ffbd P 7921060cf7c04ccf844a869c26b58c4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:20:45.008601 11382 ts_tablet_manager.cc:1905] T 96d41cf8068047a9a79a78c49fe9ffbd P 7921060cf7c04ccf844a869c26b58c4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:20:45.010380 11382 ts_tablet_manager.cc:1918] T 96d41cf8068047a9a79a78c49fe9ffbd P 7921060cf7c04ccf844a869c26b58c4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:20:45.010462 11382 log.cc:1196] T 96d41cf8068047a9a79a78c49fe9ffbd P 7921060cf7c04ccf844a869c26b58c4a: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790349610595785-3919-0/minicluster-data/ts-1/wal/wals/96d41cf8068047a9a79a78c49fe9ffbd
I20260925 15:20:45.010727 11382 ts_tablet_manager.cc:1939] T 96d41cf8068047a9a79a78c49fe9ffbd P 7921060cf7c04ccf844a869c26b58c4a: Deleting consensus metadata
I20260925 15:20:45.011514 10787 catalog_manager.cc:4841] TS 7921060cf7c04ccf844a869c26b58c4a (127.3.211.194:44803): tablet 96d41cf8068047a9a79a78c49fe9ffbd (table test-table [id=be228236955f4757be3b206f4800f523]) successfully deleted
W20260925 15:20:45.017347 11356 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:45.017157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:32847 (local address 127.0.0.1:51360)
0925 15:20:45.017215 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:45.017245 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.193:32847: connect: Broken pipe (error 32)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 15:20:45.017515 10788 catalog_manager.cc:4584] TS bf6c0d3ea9e342d6960a6f8a5d1e7e26 (127.3.211.193:32847): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 96d41cf8068047a9a79a78c49fe9ffbd: Network error: Client connection negotiation failed: client connection to 127.3.211.193:32847: connect: Broken pipe (error 32)
I20260925 15:20:45.017511 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 10990
I20260925 15:20:45.022960 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 11126
I20260925 15:20:45.027935 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 10483
I20260925 15:20:45.033084 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 10671
2026-09-25T15:20:45Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (8466 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:20:45 dist-test-slave-4bk3 krb5kdc[11386](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:20:45 dist-test-slave-4bk3 krb5kdc[11386](info): set up 2 sockets
Sep 25 15:20:45 dist-test-slave-4bk3 krb5kdc[11386](info): commencing operation
krb5kdc: starting...
W20260925 15:20:46.109409 3919 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Sep 25 15:20:46 dist-test-slave-4bk3 krb5kdc[11386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:20:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:20:46Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.211.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.211.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.211.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/krb5kdc/oryx_127.3.211.254.keytab.
Entry for principal oryx/127.3.211.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/krb5kdc/oryx_127.3.211.254.keytab.
I20260925 15:20:46.235555 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:44561
--webserver_interface=127.3.211.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/krb5kdc/oryx_127.3.211.254.keytab
--principal=oryx/127.3.211.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:35729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.211.254:44561
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:20:46.326068 11399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:46.326447 11399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:46.326475 11399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:46.328022 11399 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:20:46.328079 11399 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 15:20:46.328094 11399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:46.328109 11399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:46.328121 11399 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:20:46.328133 11399 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:20:46.329874 11399 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:35729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.211.254:44561
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/krb5kdc/oryx_127.3.211.254.keytab
--principal=oryx/127.3.211.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:44561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.211.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:46.330104 11399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:46.330287 11399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:46.335255 11407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:46.335462 11399 server_base.cc:1034] running on GCE node
W20260925 15:20:46.335278 11404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:46.335255 11405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:46.335796 11399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:46.336061 11399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:46.337222 11399 hybrid_clock.cc:648] HybridClock initialized: now 1790349646337193 us; error 39 us; skew 500 ppm
Sep 25 15:20:46 dist-test-slave-4bk3 krb5kdc[11386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349646, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.211.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:46.340184 11399 init.cc:377] Logged in from keytab as oryx/127.3.211.254@KRBTEST.COM (short username oryx)
I20260925 15:20:46.340775 11399 webserver.cc:460] Webserver started at http://127.3.211.254:46367/ using document root <none> and password file <none>
I20260925 15:20:46.341037 11399 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:46.341126 11399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:46.341288 11399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:46.342622 11399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/master-0/data/instance:
uuid: "ae6bbfe00c94422695a3080aeab48107"
format_stamp: "Formatted at 2026-09-25 15:20:46 on dist-test-slave-4bk3"
server_key: "31ef92672bada757376a59dd69932ffb"
server_key_iv: "428dc4872006d4a1023c6097061ae1d1"
server_key_version: "encryptionkey@0"
I20260925 15:20:46.343418 11399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/master-0/wal/instance:
uuid: "ae6bbfe00c94422695a3080aeab48107"
format_stamp: "Formatted at 2026-09-25 15:20:46 on dist-test-slave-4bk3"
server_key: "31ef92672bada757376a59dd69932ffb"
server_key_iv: "428dc4872006d4a1023c6097061ae1d1"
server_key_version: "encryptionkey@0"
I20260925 15:20:46.345247 11399 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260925 15:20:46.346199 11414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:46.346437 11399 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:20:46.346557 11399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/master-0/wal
uuid: "ae6bbfe00c94422695a3080aeab48107"
format_stamp: "Formatted at 2026-09-25 15:20:46 on dist-test-slave-4bk3"
server_key: "31ef92672bada757376a59dd69932ffb"
server_key_iv: "428dc4872006d4a1023c6097061ae1d1"
server_key_version: "encryptionkey@0"
I20260925 15:20:46.346649 11399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:46.357159 11399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:46.359386 11399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:46.359562 11399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:46.364652 11399 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.254:44561
I20260925 15:20:46.364779 11476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.254:44561 every 8 connection(s)
I20260925 15:20:46.365115 11399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
I20260925 15:20:46.365830 11477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:46.368676 11477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae6bbfe00c94422695a3080aeab48107: Bootstrap starting.
I20260925 15:20:46.369364 11477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae6bbfe00c94422695a3080aeab48107: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:46.369695 11477 log.cc:824] T 00000000000000000000000000000000 P ae6bbfe00c94422695a3080aeab48107: Log is configured to *not* fsync() on all Append() calls
I20260925 15:20:46.370615 11477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae6bbfe00c94422695a3080aeab48107: No bootstrap required, opened a new log
I20260925 15:20:46.371752 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 11399
I20260925 15:20:46.371871 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/master-0/wal/instance
I20260925 15:20:46.372069 3919 external_mini_cluster.cc:1428] Setting key 1bc5b84d01878d7d1d4073f743b905d1
I20260925 15:20:46.373222 11477 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae6bbfe00c94422695a3080aeab48107 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6bbfe00c94422695a3080aeab48107" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 44561 } }
I20260925 15:20:46.373414 11477 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae6bbfe00c94422695a3080aeab48107 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:46.373459 11477 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae6bbfe00c94422695a3080aeab48107 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae6bbfe00c94422695a3080aeab48107, State: Initialized, Role: FOLLOWER
I20260925 15:20:46.373605 11477 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae6bbfe00c94422695a3080aeab48107 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6bbfe00c94422695a3080aeab48107" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 44561 } }
I20260925 15:20:46.373697 11477 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae6bbfe00c94422695a3080aeab48107 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:20:46.373736 11477 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae6bbfe00c94422695a3080aeab48107 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:20:46.373790 11477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae6bbfe00c94422695a3080aeab48107 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:46.374509 11477 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae6bbfe00c94422695a3080aeab48107 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6bbfe00c94422695a3080aeab48107" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 44561 } }
I20260925 15:20:46.374637 11477 leader_election.cc:302] T 00000000000000000000000000000000 P ae6bbfe00c94422695a3080aeab48107 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae6bbfe00c94422695a3080aeab48107; no voters:
I20260925 15:20:46.374838 11477 leader_election.cc:288] T 00000000000000000000000000000000 P ae6bbfe00c94422695a3080aeab48107 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:20:46.374873 11482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae6bbfe00c94422695a3080aeab48107 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:20:46.374974 11482 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae6bbfe00c94422695a3080aeab48107 [term 1 LEADER]: Becoming Leader. State: Replica: ae6bbfe00c94422695a3080aeab48107, State: Running, Role: LEADER
I20260925 15:20:46.375119 11477 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae6bbfe00c94422695a3080aeab48107 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:20:46.375119 11482 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae6bbfe00c94422695a3080aeab48107 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6bbfe00c94422695a3080aeab48107" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 44561 } }
I20260925 15:20:46.375584 11483 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae6bbfe00c94422695a3080aeab48107 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae6bbfe00c94422695a3080aeab48107" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6bbfe00c94422695a3080aeab48107" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 44561 } } }
I20260925 15:20:46.375663 11483 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae6bbfe00c94422695a3080aeab48107 [sys.catalog]: This master's current role is: LEADER
I20260925 15:20:46.376279 11489 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:20:46.376822 11489 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:20:46.377023 11484 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae6bbfe00c94422695a3080aeab48107 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae6bbfe00c94422695a3080aeab48107. Latest consensus state: current_term: 1 leader_uuid: "ae6bbfe00c94422695a3080aeab48107" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6bbfe00c94422695a3080aeab48107" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 44561 } } }
I20260925 15:20:46.377637 11484 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae6bbfe00c94422695a3080aeab48107 [sys.catalog]: This master's current role is: LEADER
Sep 25 15:20:46 dist-test-slave-4bk3 krb5kdc[11386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349646, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM
I20260925 15:20:46.379884 11489 catalog_manager.cc:1347] Generated new cluster ID: 6df3e642d64e452fb32824f95724e8f0
I20260925 15:20:46.379935 11489 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:20:46.384229 11481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:46.372697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41334 (local address 127.3.211.254:44561)
0925 15:20:46.372877 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:46.372887 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:46.372922 (+ 35us) server_negotiation.cc:407] Connection header received
0925 15:20:46.373185 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:46.373192 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:46.373240 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:46.373361 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:46.374169 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:46.375357 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:46.376597 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:46.377200 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:46.377335 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:46.380667 (+ 3332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:46.380671 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:46.380671 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:46.380685 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:46.382519 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:46.382989 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:46.382993 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:46.382993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:46.383108 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:46.383326 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:46.383326 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:46.383327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:46.383594 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:46.383760 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:46.383867 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 15:20:46.384019 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":96,"threads_started":1}
I20260925 15:20:46.391813 11489 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:20:46.392652 11489 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:20:46.400408 11489 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae6bbfe00c94422695a3080aeab48107: Generated new TSK 0
I20260925 15:20:46.400622 11489 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.211.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.211.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.211.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/krb5kdc/oryx_127.3.211.193.keytab.
Entry for principal oryx/127.3.211.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/krb5kdc/oryx_127.3.211.193.keytab.
I20260925 15:20:46.427578 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.193:0
--local_ip_for_outbound_sockets=127.3.211.193
--webserver_interface=127.3.211.193
--webserver_port=0
--tserver_master_addrs=127.3.211.254:44561
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/krb5kdc/oryx_127.3.211.193.keytab
--principal=oryx/127.3.211.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:35729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:20:46.515041 11503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:46.515270 11503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:46.515298 11503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:46.516530 11503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:46.516573 11503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:46.516594 11503 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:20:46.516613 11503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.193
I20260925 15:20:46.518137 11503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:35729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/krb5kdc/oryx_127.3.211.193.keytab
--principal=oryx/127.3.211.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.211.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.211.254:44561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.193
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:46.518345 11503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:46.518535 11503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:46.523626 11508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:46.523638 11509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:46.523633 11511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:46.523808 11503 server_base.cc:1034] running on GCE node
I20260925 15:20:46.524056 11503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:46.524281 11503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:46.525431 11503 hybrid_clock.cc:648] HybridClock initialized: now 1790349646525417 us; error 28 us; skew 500 ppm
Sep 25 15:20:46 dist-test-slave-4bk3 krb5kdc[11386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349646, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.211.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:46.528453 11503 init.cc:377] Logged in from keytab as oryx/127.3.211.193@KRBTEST.COM (short username oryx)
I20260925 15:20:46.529022 11503 webserver.cc:460] Webserver started at http://127.3.211.193:33987/ using document root <none> and password file <none>
I20260925 15:20:46.529294 11503 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:46.529356 11503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:46.529500 11503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:46.530627 11503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-0/data/instance:
uuid: "d90226f631c24de2b88b8df73a780c0f"
format_stamp: "Formatted at 2026-09-25 15:20:46 on dist-test-slave-4bk3"
server_key: "de419ead5d99b05f8a2b8be840617105"
server_key_iv: "57cc44a415aa4424ab97328706d4a159"
server_key_version: "encryptionkey@0"
I20260925 15:20:46.531026 11503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-0/wal/instance:
uuid: "d90226f631c24de2b88b8df73a780c0f"
format_stamp: "Formatted at 2026-09-25 15:20:46 on dist-test-slave-4bk3"
server_key: "de419ead5d99b05f8a2b8be840617105"
server_key_iv: "57cc44a415aa4424ab97328706d4a159"
server_key_version: "encryptionkey@0"
I20260925 15:20:46.532660 11503 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:20:46.533577 11518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:46.533890 11503 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:46.534006 11503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-0/wal
uuid: "d90226f631c24de2b88b8df73a780c0f"
format_stamp: "Formatted at 2026-09-25 15:20:46 on dist-test-slave-4bk3"
server_key: "de419ead5d99b05f8a2b8be840617105"
server_key_iv: "57cc44a415aa4424ab97328706d4a159"
server_key_version: "encryptionkey@0"
I20260925 15:20:46.534101 11503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:46.557183 11503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:46.559607 11503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:46.559825 11503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:46.560452 11503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:20:46.560518 11503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:46.560555 11503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:20:46.560583 11503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 15:20:46 dist-test-slave-4bk3 krb5kdc[11386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349646, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.211.193@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM
I20260925 15:20:46.568461 11503 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.193:38069
I20260925 15:20:46.568544 11633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.193:38069 every 8 connection(s)
I20260925 15:20:46.568929 11503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:20:46.572537 11481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:46.561039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:56421 (local address 127.3.211.254:44561)
0925 15:20:46.561121 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:46.561123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:46.561337 (+ 214us) server_negotiation.cc:407] Connection header received
0925 15:20:46.561908 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:46.561909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:46.561925 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:46.561984 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:46.563051 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:46.563638 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:46.565022 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:46.565255 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:46.565342 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:46.568809 (+ 3467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:46.568811 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:46.568812 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:46.568821 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:46.570745 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:46.571327 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:46.571329 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:46.571329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:46.571416 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:46.571733 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:46.571734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:46.571734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:46.571899 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:46.571977 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:46.572282 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 15:20:46.572413 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 15:20:46.572796 11527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:46.560997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:44561 (local address 127.3.211.193:56421)
0925 15:20:46.561210 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:46.561247 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:46.561776 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:46.562061 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:46.562087 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:46.562533 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:46.562945 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:46.562953 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:46.563749 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:46.563750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:46.564915 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:46.564919 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:46.565363 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:46.565364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:46.565538 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:46.566187 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:46.566201 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:46.568640 (+ 2439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:46.570890 (+ 2250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:46.570909 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:46.570911 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:46.571204 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:46.571493 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:46.571493 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:46.571494 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:46.571617 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:46.571979 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:46.571987 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:46.572142 (+ 155us) client_negotiation.cc:769] Sending connection context
0925 15:20:46.572259 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 15:20:46.572526 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":90,"threads_started":1}
I20260925 15:20:46.573256 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 11503
I20260925 15:20:46.573387 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-0/wal/instance
I20260925 15:20:46.573405 11634 heartbeater.cc:347] Connected to a master server at 127.3.211.254:44561
I20260925 15:20:46.573509 11634 heartbeater.cc:464] Registering TS with master...
I20260925 15:20:46.573643 3919 external_mini_cluster.cc:1428] Setting key f46bb48777b39a75a001a1c26a4b5b2f
I20260925 15:20:46.573742 11634 heartbeater.cc:511] Master 127.3.211.254:44561 requested a full tablet report, sending...
I20260925 15:20:46.574249 11431 ts_manager.cc:194] Registered new tserver with Master: d90226f631c24de2b88b8df73a780c0f (127.3.211.193:38069)
I20260925 15:20:46.575380 11431 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.211.193@KRBTEST.COM'} at 127.3.211.193:56421
I20260925 15:20:46.581936 11640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:46.574512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:44561 (local address 127.3.211.193:45479)
0925 15:20:46.574641 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:46.574652 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:46.574744 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:46.574993 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:46.574994 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:46.575219 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:46.575294 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:46.575296 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:46.576319 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:46.576320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:46.577197 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:46.577200 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:46.577577 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:46.577579 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:46.577674 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:46.578247 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:46.578252 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:46.578557 (+ 305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:46.580786 (+ 2229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:46.580787 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:46.580788 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:46.581020 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:46.581253 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:46.581254 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:46.581254 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:46.581315 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:46.581622 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:46.581623 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:46.581674 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 15:20:46.581754 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 15:20:46.581857 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":64,"threads_started":1}
I20260925 15:20:46.582003 11481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:46.574824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:45479 (local address 127.3.211.254:44561)
0925 15:20:46.574868 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:46.574870 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:46.574877 (+ 7us) server_negotiation.cc:407] Connection header received
0925 15:20:46.574898 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:46.574899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:46.574909 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:46.574953 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:46.575376 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:46.576210 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:46.577302 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:46.577494 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:46.577556 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:46.578659 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:46.578660 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:46.578661 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:46.578669 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:46.580669 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:46.581108 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:46.581110 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:46.581110 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:46.581178 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:46.581384 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:46.581385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:46.581385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:46.581542 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:46.581616 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:46.581789 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 15:20:46.581896 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
WARNING: no policy specified for oryx/127.3.211.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.211.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.211.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/krb5kdc/oryx_127.3.211.194.keytab.
Entry for principal oryx/127.3.211.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/krb5kdc/oryx_127.3.211.194.keytab.
I20260925 15:20:46.604125 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.194:0
--local_ip_for_outbound_sockets=127.3.211.194
--webserver_interface=127.3.211.194
--webserver_port=0
--tserver_master_addrs=127.3.211.254:44561
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/krb5kdc/oryx_127.3.211.194.keytab
--principal=oryx/127.3.211.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:35729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:20:46.691588 11643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:46.691826 11643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:46.691854 11643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:46.693051 11643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:46.693099 11643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:46.693121 11643 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:20:46.693140 11643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.194
I20260925 15:20:46.694707 11643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:35729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/krb5kdc/oryx_127.3.211.194.keytab
--principal=oryx/127.3.211.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.211.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.211.254:44561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.194
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:46.694936 11643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:46.695130 11643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:46.700253 11651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:46.700361 11649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:46.700253 11648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:46.700747 11643 server_base.cc:1034] running on GCE node
I20260925 15:20:46.701018 11643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:46.701267 11643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:46.702440 11643 hybrid_clock.cc:648] HybridClock initialized: now 1790349646702416 us; error 42 us; skew 500 ppm
Sep 25 15:20:46 dist-test-slave-4bk3 krb5kdc[11386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349646, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.211.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:46.705264 11643 init.cc:377] Logged in from keytab as oryx/127.3.211.194@KRBTEST.COM (short username oryx)
I20260925 15:20:46.705785 11643 webserver.cc:460] Webserver started at http://127.3.211.194:39993/ using document root <none> and password file <none>
I20260925 15:20:46.706043 11643 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:46.706115 11643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:46.706269 11643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:46.707369 11643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-1/data/instance:
uuid: "64dcb0cfa5be4e0291110ab0fd85c27f"
format_stamp: "Formatted at 2026-09-25 15:20:46 on dist-test-slave-4bk3"
server_key: "918e07539cdbf94279fa0d8e6da96b33"
server_key_iv: "239a900ba6916303630a83e60ceadb3f"
server_key_version: "encryptionkey@0"
I20260925 15:20:46.707751 11643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-1/wal/instance:
uuid: "64dcb0cfa5be4e0291110ab0fd85c27f"
format_stamp: "Formatted at 2026-09-25 15:20:46 on dist-test-slave-4bk3"
server_key: "918e07539cdbf94279fa0d8e6da96b33"
server_key_iv: "239a900ba6916303630a83e60ceadb3f"
server_key_version: "encryptionkey@0"
I20260925 15:20:46.709350 11643 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:20:46.710290 11658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:46.710629 11643 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:46.710744 11643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-1/wal
uuid: "64dcb0cfa5be4e0291110ab0fd85c27f"
format_stamp: "Formatted at 2026-09-25 15:20:46 on dist-test-slave-4bk3"
server_key: "918e07539cdbf94279fa0d8e6da96b33"
server_key_iv: "239a900ba6916303630a83e60ceadb3f"
server_key_version: "encryptionkey@0"
I20260925 15:20:46.710829 11643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:46.734903 11643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:46.737546 11643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:46.737722 11643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:46.738247 11643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:20:46.738313 11643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:46.738360 11643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:20:46.738387 11643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:46.746122 11643 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.194:44601
I20260925 15:20:46.746203 11773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.194:44601 every 8 connection(s)
Sep 25 15:20:46 dist-test-slave-4bk3 krb5kdc[11386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349646, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.211.194@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM
I20260925 15:20:46.746522 11643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
I20260925 15:20:46.750535 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 11643
I20260925 15:20:46.750674 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-1/wal/instance
I20260925 15:20:46.750914 3919 external_mini_cluster.cc:1428] Setting key bba42d79b6f1d36853d027a447834119
I20260925 15:20:46.751184 11481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:46.738792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:52853 (local address 127.3.211.254:44561)
0925 15:20:46.738887 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:46.738890 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:46.739062 (+ 172us) server_negotiation.cc:407] Connection header received
0925 15:20:46.739546 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:46.739548 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:46.739562 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:46.739649 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:46.740623 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:46.741477 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:46.743193 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:46.743582 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:46.743668 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:46.747107 (+ 3439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:46.747109 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:46.747109 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:46.747118 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:46.749314 (+ 2196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:46.749879 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:46.749881 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:46.749881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:46.749961 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:46.750263 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:46.750264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:46.750265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:46.750453 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:46.750544 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:46.750972 (+ 428us) server_negotiation.cc:299] Negotiation successful
0925 15:20:46.751079 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 15:20:46.751431 11667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:46.738759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:44561 (local address 127.3.211.194:52853)
0925 15:20:46.738927 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:46.738968 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:46.739448 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:46.739749 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:46.739754 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:46.740212 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:46.740508 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:46.740513 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:46.741583 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:46.741583 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:46.743096 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:46.743099 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:46.743715 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:46.743716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:46.743905 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:46.744836 (+ 931us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:46.744849 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:46.746997 (+ 2148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:46.749461 (+ 2464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:46.749474 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:46.749477 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:46.749778 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:46.750026 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:46.750028 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:46.750028 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:46.750166 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:46.750556 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:46.750563 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:46.750807 (+ 244us) client_negotiation.cc:769] Sending connection context
0925 15:20:46.750919 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 15:20:46.751178 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":82,"threads_started":1}
I20260925 15:20:46.752270 11774 heartbeater.cc:347] Connected to a master server at 127.3.211.254:44561
I20260925 15:20:46.752403 11774 heartbeater.cc:464] Registering TS with master...
I20260925 15:20:46.752609 11774 heartbeater.cc:511] Master 127.3.211.254:44561 requested a full tablet report, sending...
I20260925 15:20:46.753005 11431 ts_manager.cc:194] Registered new tserver with Master: 64dcb0cfa5be4e0291110ab0fd85c27f (127.3.211.194:44601)
I20260925 15:20:46.753541 11431 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.211.194@KRBTEST.COM'} at 127.3.211.194:52853
I20260925 15:20:46.761037 11481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:46.753483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:51197 (local address 127.3.211.254:44561)
0925 15:20:46.753530 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:46.753532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:46.753784 (+ 252us) server_negotiation.cc:407] Connection header received
0925 15:20:46.753848 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:46.753849 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:46.753861 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:46.753923 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:46.754314 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:46.755104 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:46.756399 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:46.756608 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:46.756672 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:46.757860 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:46.757861 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:46.757862 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:46.757869 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:46.759574 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:46.760060 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:46.760061 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:46.760061 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:46.760122 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:46.760412 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:46.760413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:46.760413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:46.760587 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:46.760646 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:46.760883 (+ 237us) server_negotiation.cc:299] Negotiation successful
0925 15:20:46.760967 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 15:20:46.761185 11781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:46.753566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:44561 (local address 127.3.211.194:51197)
0925 15:20:46.753717 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:46.753728 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:46.753794 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:46.753929 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:46.753930 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:46.754147 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:46.754218 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:46.754220 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:46.755239 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:46.755240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:46.756207 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:46.756211 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:46.756702 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:46.756704 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:46.756779 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:46.757217 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:46.757223 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:46.757710 (+ 487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:46.759706 (+ 1996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:46.759708 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:46.759709 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:46.759949 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:46.760208 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:46.760209 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:46.760209 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:46.760285 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:46.760665 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:46.760666 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:46.760736 (+ 70us) client_negotiation.cc:769] Sending connection context
0925 15:20:46.760825 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 15:20:46.761049 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":80,"threads_started":1}
WARNING: no policy specified for oryx/127.3.211.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.211.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.211.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/krb5kdc/oryx_127.3.211.195.keytab.
Entry for principal oryx/127.3.211.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/krb5kdc/oryx_127.3.211.195.keytab.
I20260925 15:20:46.779435 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.195:0
--local_ip_for_outbound_sockets=127.3.211.195
--webserver_interface=127.3.211.195
--webserver_port=0
--tserver_master_addrs=127.3.211.254:44561
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/krb5kdc/oryx_127.3.211.195.keytab
--principal=oryx/127.3.211.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:35729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:20:46.864120 11783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:46.864359 11783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:46.864434 11783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:46.865688 11783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:46.865732 11783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:46.865754 11783 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:20:46.865806 11783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.195
I20260925 15:20:46.867393 11783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:35729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/krb5kdc/oryx_127.3.211.195.keytab
--principal=oryx/127.3.211.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.211.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.211.254:44561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.195
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:46.867645 11783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:46.867897 11783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:46.873250 11791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:46.873224 11789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:46.873224 11788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:46.873252 11783 server_base.cc:1034] running on GCE node
I20260925 15:20:46.873656 11783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:46.873950 11783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:46.875108 11783 hybrid_clock.cc:648] HybridClock initialized: now 1790349646875094 us; error 45 us; skew 500 ppm
Sep 25 15:20:46 dist-test-slave-4bk3 krb5kdc[11386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349646, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.211.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:46.877941 11783 init.cc:377] Logged in from keytab as oryx/127.3.211.195@KRBTEST.COM (short username oryx)
I20260925 15:20:46.878436 11783 webserver.cc:460] Webserver started at http://127.3.211.195:37297/ using document root <none> and password file <none>
I20260925 15:20:46.878674 11783 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:46.878723 11783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:46.878875 11783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:46.879988 11783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-2/data/instance:
uuid: "1d5976b8e2a84424b64a34b9e87adf95"
format_stamp: "Formatted at 2026-09-25 15:20:46 on dist-test-slave-4bk3"
server_key: "8b72600d02cbe3a11e5e42acd90b26c1"
server_key_iv: "c88e59bd77381d9a751f91e0268fd6df"
server_key_version: "encryptionkey@0"
I20260925 15:20:46.880427 11783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-2/wal/instance:
uuid: "1d5976b8e2a84424b64a34b9e87adf95"
format_stamp: "Formatted at 2026-09-25 15:20:46 on dist-test-slave-4bk3"
server_key: "8b72600d02cbe3a11e5e42acd90b26c1"
server_key_iv: "c88e59bd77381d9a751f91e0268fd6df"
server_key_version: "encryptionkey@0"
I20260925 15:20:46.881934 11783 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:46.882818 11798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:46.883116 11783 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:46.883249 11783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-2/wal
uuid: "1d5976b8e2a84424b64a34b9e87adf95"
format_stamp: "Formatted at 2026-09-25 15:20:46 on dist-test-slave-4bk3"
server_key: "8b72600d02cbe3a11e5e42acd90b26c1"
server_key_iv: "c88e59bd77381d9a751f91e0268fd6df"
server_key_version: "encryptionkey@0"
I20260925 15:20:46.883342 11783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:46.891184 11783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:46.893603 11783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:46.893782 11783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:46.894306 11783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:20:46.894375 11783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:46.894419 11783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:20:46.894449 11783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:46.902348 11783 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.195:41405
I20260925 15:20:46.902431 11913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.195:41405 every 8 connection(s)
I20260925 15:20:46.902765 11783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
Sep 25 15:20:46 dist-test-slave-4bk3 krb5kdc[11386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349646, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.211.195@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM
I20260925 15:20:46.904623 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 11783
I20260925 15:20:46.904733 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-2/wal/instance
I20260925 15:20:46.904920 3919 external_mini_cluster.cc:1428] Setting key a1584a2728e1c98b34746886f3210ceb
I20260925 15:20:46.906759 11481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:46.894857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:34347 (local address 127.3.211.254:44561)
0925 15:20:46.894929 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:46.894932 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:46.895216 (+ 284us) server_negotiation.cc:407] Connection header received
0925 15:20:46.896024 (+ 808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:46.896026 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:46.896049 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:46.896134 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:46.897163 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:46.898034 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:46.899884 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:46.900192 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:46.900295 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:46.903527 (+ 3232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:46.903529 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:46.903529 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:46.903537 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:46.905201 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:46.905756 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:46.905758 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:46.905758 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:46.905822 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:46.906088 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:46.906089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:46.906089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:46.906219 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:46.906296 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:46.906579 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 15:20:46.906664 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 15:20:46.906985 11807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:46.894842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:44561 (local address 127.3.211.195:34347)
0925 15:20:46.895029 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:46.895081 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:46.895845 (+ 764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:46.896259 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:46.896264 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:46.896711 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:46.897037 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:46.897044 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:46.898167 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:46.898168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:46.899753 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:46.899758 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:46.900310 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:46.900311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:46.900552 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:46.901251 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:46.901263 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:46.903389 (+ 2126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:46.905294 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:46.905305 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:46.905308 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:46.905669 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:46.905894 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:46.905895 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:46.905895 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:46.906019 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:46.906287 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:46.906293 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:46.906441 (+ 148us) client_negotiation.cc:769] Sending connection context
0925 15:20:46.906556 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 15:20:46.906727 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":93,"threads_started":1}
I20260925 15:20:46.907513 11914 heartbeater.cc:347] Connected to a master server at 127.3.211.254:44561
I20260925 15:20:46.907605 11914 heartbeater.cc:464] Registering TS with master...
I20260925 15:20:46.907806 11914 heartbeater.cc:511] Master 127.3.211.254:44561 requested a full tablet report, sending...
I20260925 15:20:46.908238 11430 ts_manager.cc:194] Registered new tserver with Master: 1d5976b8e2a84424b64a34b9e87adf95 (127.3.211.195:41405)
I20260925 15:20:46.908789 11430 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.211.195@KRBTEST.COM'} at 127.3.211.195:34347
I20260925 15:20:46.914036 11481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:46.908660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:33621 (local address 127.3.211.254:44561)
0925 15:20:46.908705 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:46.908707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:46.908773 (+ 66us) server_negotiation.cc:407] Connection header received
0925 15:20:46.908844 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:46.908845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:46.908871 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:46.908930 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:46.909360 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:46.909809 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:46.910492 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:46.910681 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:46.910735 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:46.911577 (+ 842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:46.911578 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:46.911578 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:46.911584 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:46.912848 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:46.913210 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:46.913211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:46.913211 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:46.913276 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:46.913492 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:46.913493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:46.913493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:46.913610 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:46.913685 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:46.913837 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 15:20:46.913930 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 15:20:46.914057 11920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:46.908571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:44561 (local address 127.3.211.195:33621)
0925 15:20:46.908715 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:46.908726 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:46.908786 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:46.908980 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:46.908981 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:46.909204 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:46.909274 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:46.909277 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:46.909911 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:46.909912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:46.910415 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:46.910417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:46.910747 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:46.910747 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:46.910796 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:46.911200 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:46.911205 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:46.911496 (+ 291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:46.912925 (+ 1429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:46.912925 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:46.912926 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:46.913102 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:46.913357 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:46.913358 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:46.913359 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:46.913423 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:46.913688 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:46.913689 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:46.913744 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 15:20:46.913812 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 15:20:46.913932 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":94,"threads_started":1}
I20260925 15:20:46.916850 3919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 15:20:46 dist-test-slave-4bk3 krb5kdc[11386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349646, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 15:20:46 dist-test-slave-4bk3 krb5kdc[11386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:46 dist-test-slave-4bk3 krb5kdc[11386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 15:20:46.940100 11481 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:46.933697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41344 (local address 127.3.211.254:44561)
0925 15:20:46.933796 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:46.933798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:46.933807 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:20:46.933833 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:46.933834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:46.933843 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:46.933903 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:46.934627 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:46.935133 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:46.935995 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:46.936276 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:46.936411 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:46.940005 (+ 3594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41344: BlockingRecv error: recv got EOF from 127.0.0.1:41344 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 25 15:20:46 dist-test-slave-4bk3 krb5kdc[11386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349646, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM
I20260925 15:20:46.948428 11481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:46.940983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41356 (local address 127.3.211.254:44561)
0925 15:20:46.941073 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:46.941075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:46.941233 (+ 158us) server_negotiation.cc:407] Connection header received
0925 15:20:46.941376 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:46.941377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:46.941389 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:46.941458 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:46.941987 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:46.942481 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:46.943191 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:46.943398 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:46.943488 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:46.945949 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:46.945950 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:46.945950 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:46.945959 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:46.947250 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:46.947618 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:46.947619 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:46.947620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:46.947675 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:46.947872 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:46.947873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:46.947873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:46.947991 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:46.948099 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:46.948206 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 15:20:46.948289 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 15:20:46.950098 11430 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41356:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:20:46.951761 11430 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:20:46.958643 11941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:46.954764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40926 (local address 127.3.211.195:41405)
0925 15:20:46.954936 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:46.954940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:46.954950 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:20:46.954982 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:46.954985 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:46.954998 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:46.955076 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:20:46.955327 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:46.956415 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:46.957470 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:46.958185 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:46.958268 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:46.958345 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:46.958486 (+ 141us) server_negotiation.cc:299] Negotiation successful
0925 15:20:46.958531 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":79,"threads_started":1}
I20260925 15:20:46.958645 11936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:46.954565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.195:41405 (local address 127.0.0.1:40926)
0925 15:20:46.954767 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:46.954775 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:46.954811 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:46.955218 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:46.955218 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:46.955220 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:20:46.955249 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:46.955251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:46.956521 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:46.956523 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:46.957377 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:46.957379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:46.958290 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:46.958291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:46.958365 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:46.958378 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 15:20:46.958452 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 15:20:46.958539 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":79,"threads_started":1}
I20260925 15:20:46.959064 11937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:46.954334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:38069 (local address 127.0.0.1:53252)
0925 15:20:46.954535 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:46.954576 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:46.954645 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:46.954937 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:46.954940 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:46.954943 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:20:46.955021 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:46.955023 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:46.956494 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:46.956495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:46.957584 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:46.957587 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:46.958520 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:46.958521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:46.958572 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:46.958576 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:20:46.958630 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 15:20:46.958668 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":7,"thread_start_us":85,"threads_started":1}
I20260925 15:20:46.959280 11938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:46.954439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53252 (local address 127.3.211.193:38069)
0925 15:20:46.954626 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:46.954632 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:46.954645 (+ 13us) server_negotiation.cc:407] Connection header received
0925 15:20:46.954745 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:46.954747 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:46.954763 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:46.954851 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:20:46.955202 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:46.956407 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:46.957677 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:46.958408 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:46.958498 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:46.958565 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:46.958731 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 15:20:46.959102 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":78,"threads_started":1}
I20260925 15:20:46.959441 11848 tablet_service.cc:1467] Processing CreateTablet for tablet 2878733a76da4e3680597428f3b3b20a (DEFAULT_TABLE table=test-table [id=1de5f1797baa42ba8d4fabfb3e9a2d14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:20:46.959872 11848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2878733a76da4e3680597428f3b3b20a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:46.959957 11940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:46.954348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.194:44601 (local address 127.0.0.1:55894)
0925 15:20:46.954697 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:46.954716 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:46.954761 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:46.955092 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:46.955093 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:46.955094 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:20:46.955137 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:46.955138 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:46.956157 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:46.956158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:46.957584 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:46.957587 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:46.959670 (+ 2083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:46.959671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:46.959762 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:46.959767 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:20:46.959837 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 15:20:46.959881 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":30,"thread_start_us":88,"threads_started":1}
I20260925 15:20:46.959996 11939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:46.954436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55894 (local address 127.3.211.194:44601)
0925 15:20:46.954920 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:46.954925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:46.954935 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:20:46.954975 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:46.954977 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:46.954993 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:46.955075 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:20:46.955199 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:46.956061 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:46.958606 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:46.959573 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:46.959670 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:46.959742 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:46.959853 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 15:20:46.959899 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":92,"threads_started":1}
I20260925 15:20:46.959913 11568 tablet_service.cc:1467] Processing CreateTablet for tablet 2878733a76da4e3680597428f3b3b20a (DEFAULT_TABLE table=test-table [id=1de5f1797baa42ba8d4fabfb3e9a2d14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:20:46.960289 11568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2878733a76da4e3680597428f3b3b20a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:46.960598 11708 tablet_service.cc:1467] Processing CreateTablet for tablet 2878733a76da4e3680597428f3b3b20a (DEFAULT_TABLE table=test-table [id=1de5f1797baa42ba8d4fabfb3e9a2d14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:20:46.960932 11708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2878733a76da4e3680597428f3b3b20a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:46.964865 11942 tablet_bootstrap.cc:492] T 2878733a76da4e3680597428f3b3b20a P 1d5976b8e2a84424b64a34b9e87adf95: Bootstrap starting.
I20260925 15:20:46.965348 11944 tablet_bootstrap.cc:492] T 2878733a76da4e3680597428f3b3b20a P 64dcb0cfa5be4e0291110ab0fd85c27f: Bootstrap starting.
I20260925 15:20:46.965536 11942 tablet_bootstrap.cc:654] T 2878733a76da4e3680597428f3b3b20a P 1d5976b8e2a84424b64a34b9e87adf95: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:46.965547 11943 tablet_bootstrap.cc:492] T 2878733a76da4e3680597428f3b3b20a P d90226f631c24de2b88b8df73a780c0f: Bootstrap starting.
I20260925 15:20:46.965850 11942 log.cc:824] T 2878733a76da4e3680597428f3b3b20a P 1d5976b8e2a84424b64a34b9e87adf95: Log is configured to *not* fsync() on all Append() calls
I20260925 15:20:46.965981 11944 tablet_bootstrap.cc:654] T 2878733a76da4e3680597428f3b3b20a P 64dcb0cfa5be4e0291110ab0fd85c27f: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:46.966161 11943 tablet_bootstrap.cc:654] T 2878733a76da4e3680597428f3b3b20a P d90226f631c24de2b88b8df73a780c0f: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:46.966369 11944 log.cc:824] T 2878733a76da4e3680597428f3b3b20a P 64dcb0cfa5be4e0291110ab0fd85c27f: Log is configured to *not* fsync() on all Append() calls
I20260925 15:20:46.966495 11943 log.cc:824] T 2878733a76da4e3680597428f3b3b20a P d90226f631c24de2b88b8df73a780c0f: Log is configured to *not* fsync() on all Append() calls
I20260925 15:20:46.966702 11942 tablet_bootstrap.cc:492] T 2878733a76da4e3680597428f3b3b20a P 1d5976b8e2a84424b64a34b9e87adf95: No bootstrap required, opened a new log
I20260925 15:20:46.966796 11942 ts_tablet_manager.cc:1397] T 2878733a76da4e3680597428f3b3b20a P 1d5976b8e2a84424b64a34b9e87adf95: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:20:46.967321 11943 tablet_bootstrap.cc:492] T 2878733a76da4e3680597428f3b3b20a P d90226f631c24de2b88b8df73a780c0f: No bootstrap required, opened a new log
I20260925 15:20:46.967394 11943 ts_tablet_manager.cc:1397] T 2878733a76da4e3680597428f3b3b20a P d90226f631c24de2b88b8df73a780c0f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:20:46.968842 11944 tablet_bootstrap.cc:492] T 2878733a76da4e3680597428f3b3b20a P 64dcb0cfa5be4e0291110ab0fd85c27f: No bootstrap required, opened a new log
I20260925 15:20:46.968941 11944 ts_tablet_manager.cc:1397] T 2878733a76da4e3680597428f3b3b20a P 64dcb0cfa5be4e0291110ab0fd85c27f: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260925 15:20:46.969027 11942 raft_consensus.cc:348] T 2878733a76da4e3680597428f3b3b20a P 1d5976b8e2a84424b64a34b9e87adf95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } }
I20260925 15:20:46.969231 11942 raft_consensus.cc:374] T 2878733a76da4e3680597428f3b3b20a P 1d5976b8e2a84424b64a34b9e87adf95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:46.969267 11942 raft_consensus.cc:729] T 2878733a76da4e3680597428f3b3b20a P 1d5976b8e2a84424b64a34b9e87adf95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d5976b8e2a84424b64a34b9e87adf95, State: Initialized, Role: FOLLOWER
I20260925 15:20:46.969396 11942 consensus_queue.cc:260] T 2878733a76da4e3680597428f3b3b20a P 1d5976b8e2a84424b64a34b9e87adf95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } }
I20260925 15:20:46.969470 11943 raft_consensus.cc:348] T 2878733a76da4e3680597428f3b3b20a P d90226f631c24de2b88b8df73a780c0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } }
I20260925 15:20:46.969631 11943 raft_consensus.cc:374] T 2878733a76da4e3680597428f3b3b20a P d90226f631c24de2b88b8df73a780c0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:46.969683 11943 raft_consensus.cc:729] T 2878733a76da4e3680597428f3b3b20a P d90226f631c24de2b88b8df73a780c0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d90226f631c24de2b88b8df73a780c0f, State: Initialized, Role: FOLLOWER
I20260925 15:20:46.969702 11942 ts_tablet_manager.cc:1428] T 2878733a76da4e3680597428f3b3b20a P 1d5976b8e2a84424b64a34b9e87adf95: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:20:46.969803 11943 consensus_queue.cc:260] T 2878733a76da4e3680597428f3b3b20a P d90226f631c24de2b88b8df73a780c0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } }
I20260925 15:20:46.969863 11914 heartbeater.cc:503] Master 127.3.211.254:44561 was elected leader, sending a full tablet report...
I20260925 15:20:46.970101 11943 ts_tablet_manager.cc:1428] T 2878733a76da4e3680597428f3b3b20a P d90226f631c24de2b88b8df73a780c0f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:20:46.970146 11634 heartbeater.cc:503] Master 127.3.211.254:44561 was elected leader, sending a full tablet report...
I20260925 15:20:46.971021 11944 raft_consensus.cc:348] T 2878733a76da4e3680597428f3b3b20a P 64dcb0cfa5be4e0291110ab0fd85c27f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } }
I20260925 15:20:46.971205 11944 raft_consensus.cc:374] T 2878733a76da4e3680597428f3b3b20a P 64dcb0cfa5be4e0291110ab0fd85c27f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:46.971252 11944 raft_consensus.cc:729] T 2878733a76da4e3680597428f3b3b20a P 64dcb0cfa5be4e0291110ab0fd85c27f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64dcb0cfa5be4e0291110ab0fd85c27f, State: Initialized, Role: FOLLOWER
I20260925 15:20:46.971387 11944 consensus_queue.cc:260] T 2878733a76da4e3680597428f3b3b20a P 64dcb0cfa5be4e0291110ab0fd85c27f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } }
I20260925 15:20:46.971670 11944 ts_tablet_manager.cc:1428] T 2878733a76da4e3680597428f3b3b20a P 64dcb0cfa5be4e0291110ab0fd85c27f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:20:46.971697 11774 heartbeater.cc:503] Master 127.3.211.254:44561 was elected leader, sending a full tablet report...
I20260925 15:20:46.980062 11950 raft_consensus.cc:482] T 2878733a76da4e3680597428f3b3b20a P 64dcb0cfa5be4e0291110ab0fd85c27f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:20:46.980190 11950 raft_consensus.cc:504] T 2878733a76da4e3680597428f3b3b20a P 64dcb0cfa5be4e0291110ab0fd85c27f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } }
I20260925 15:20:46.980597 11950 leader_election.cc:288] T 2878733a76da4e3680597428f3b3b20a P 64dcb0cfa5be4e0291110ab0fd85c27f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d90226f631c24de2b88b8df73a780c0f (127.3.211.193:38069), 1d5976b8e2a84424b64a34b9e87adf95 (127.3.211.195:41405)
I20260925 15:20:46.984975 11938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:46.980947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:34641 (local address 127.3.211.193:38069)
0925 15:20:46.981004 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:46.981006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:46.981017 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:20:46.981085 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:46.981086 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:46.981096 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:46.981170 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:20:46.981386 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:46.982147 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:46.983622 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:46.984258 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:46.984403 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:46.984412 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:46.984857 (+ 445us) server_negotiation.cc:299] Negotiation successful
0925 15:20:46.984917 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 15:20:46.984957 11941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:46.981240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:52629 (local address 127.3.211.195:41405)
0925 15:20:46.981313 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:46.981315 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:46.981376 (+ 61us) server_negotiation.cc:407] Connection header received
0925 15:20:46.981444 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:46.981445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:46.981455 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:46.981501 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:20:46.981672 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:46.982399 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:46.983716 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:46.984269 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:46.984381 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:46.984393 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:46.984747 (+ 354us) server_negotiation.cc:299] Negotiation successful
0925 15:20:46.984877 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 15:20:46.985073 11951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:46.981014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.195:41405 (local address 127.3.211.194:52629)
0925 15:20:46.981320 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:46.981330 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:46.981392 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:46.981514 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:46.981514 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:46.981516 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:20:46.981594 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:46.981595 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:46.982501 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:46.982502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:46.983623 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:46.983625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:46.984403 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:46.984404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:46.984605 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:46.984611 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 15:20:46.984771 (+ 160us) client_negotiation.cc:240] Negotiation successful
0925 15:20:46.984835 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":116,"threads_started":1}
I20260925 15:20:46.985077 11667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:46.980899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:38069 (local address 127.3.211.194:34641)
0925 15:20:46.980945 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:46.980960 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:46.981010 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:46.981187 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:46.981188 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:46.981191 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:20:46.981286 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:46.981289 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:46.982243 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:46.982244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:46.983466 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:46.983469 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:46.984402 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:46.984404 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:46.984615 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:46.984618 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:20:46.984797 (+ 179us) client_negotiation.cc:240] Negotiation successful
0925 15:20:46.984835 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23,"mutex_wait_us":3}
I20260925 15:20:46.986214 11588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2878733a76da4e3680597428f3b3b20a" candidate_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d90226f631c24de2b88b8df73a780c0f" is_pre_election: true
I20260925 15:20:46.986218 11868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2878733a76da4e3680597428f3b3b20a" candidate_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d5976b8e2a84424b64a34b9e87adf95" is_pre_election: true
I20260925 15:20:46.986423 11588 raft_consensus.cc:2454] T 2878733a76da4e3680597428f3b3b20a P d90226f631c24de2b88b8df73a780c0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64dcb0cfa5be4e0291110ab0fd85c27f in term 0.
I20260925 15:20:46.986423 11868 raft_consensus.cc:2454] T 2878733a76da4e3680597428f3b3b20a P 1d5976b8e2a84424b64a34b9e87adf95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64dcb0cfa5be4e0291110ab0fd85c27f in term 0.
I20260925 15:20:46.986693 11659 leader_election.cc:302] T 2878733a76da4e3680597428f3b3b20a P 64dcb0cfa5be4e0291110ab0fd85c27f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64dcb0cfa5be4e0291110ab0fd85c27f, d90226f631c24de2b88b8df73a780c0f; no voters:
I20260925 15:20:46.986976 11950 raft_consensus.cc:2781] T 2878733a76da4e3680597428f3b3b20a P 64dcb0cfa5be4e0291110ab0fd85c27f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:20:46.987071 11950 raft_consensus.cc:482] T 2878733a76da4e3680597428f3b3b20a P 64dcb0cfa5be4e0291110ab0fd85c27f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:20:46.987092 11950 raft_consensus.cc:3037] T 2878733a76da4e3680597428f3b3b20a P 64dcb0cfa5be4e0291110ab0fd85c27f [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:46.988322 11950 raft_consensus.cc:504] T 2878733a76da4e3680597428f3b3b20a P 64dcb0cfa5be4e0291110ab0fd85c27f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } }
I20260925 15:20:46.988562 11950 leader_election.cc:288] T 2878733a76da4e3680597428f3b3b20a P 64dcb0cfa5be4e0291110ab0fd85c27f [CANDIDATE]: Term 1 election: Requested vote from peers d90226f631c24de2b88b8df73a780c0f (127.3.211.193:38069), 1d5976b8e2a84424b64a34b9e87adf95 (127.3.211.195:41405)
I20260925 15:20:46.988807 11868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2878733a76da4e3680597428f3b3b20a" candidate_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d5976b8e2a84424b64a34b9e87adf95"
I20260925 15:20:46.988807 11588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2878733a76da4e3680597428f3b3b20a" candidate_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d90226f631c24de2b88b8df73a780c0f"
I20260925 15:20:46.988905 11868 raft_consensus.cc:3037] T 2878733a76da4e3680597428f3b3b20a P 1d5976b8e2a84424b64a34b9e87adf95 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:46.988906 11588 raft_consensus.cc:3037] T 2878733a76da4e3680597428f3b3b20a P d90226f631c24de2b88b8df73a780c0f [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:46.989820 11868 raft_consensus.cc:2454] T 2878733a76da4e3680597428f3b3b20a P 1d5976b8e2a84424b64a34b9e87adf95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64dcb0cfa5be4e0291110ab0fd85c27f in term 1.
I20260925 15:20:46.989820 11588 raft_consensus.cc:2454] T 2878733a76da4e3680597428f3b3b20a P d90226f631c24de2b88b8df73a780c0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64dcb0cfa5be4e0291110ab0fd85c27f in term 1.
I20260925 15:20:46.990048 11659 leader_election.cc:302] T 2878733a76da4e3680597428f3b3b20a P 64dcb0cfa5be4e0291110ab0fd85c27f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d5976b8e2a84424b64a34b9e87adf95, 64dcb0cfa5be4e0291110ab0fd85c27f; no voters:
I20260925 15:20:46.990154 11950 raft_consensus.cc:2781] T 2878733a76da4e3680597428f3b3b20a P 64dcb0cfa5be4e0291110ab0fd85c27f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:20:46.990393 11950 raft_consensus.cc:686] T 2878733a76da4e3680597428f3b3b20a P 64dcb0cfa5be4e0291110ab0fd85c27f [term 1 LEADER]: Becoming Leader. State: Replica: 64dcb0cfa5be4e0291110ab0fd85c27f, State: Running, Role: LEADER
I20260925 15:20:46.990520 11950 consensus_queue.cc:237] T 2878733a76da4e3680597428f3b3b20a P 64dcb0cfa5be4e0291110ab0fd85c27f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } }
I20260925 15:20:46.991405 11431 catalog_manager.cc:5495] T 2878733a76da4e3680597428f3b3b20a P 64dcb0cfa5be4e0291110ab0fd85c27f reported cstate change: term changed from 0 to 1, leader changed from <none> to 64dcb0cfa5be4e0291110ab0fd85c27f (127.3.211.194). New cstate: current_term: 1 leader_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } health_report { overall_health: UNKNOWN } } }
W20260925 15:20:46.997184 11775 tablet.cc:2369] T 2878733a76da4e3680597428f3b3b20a P 64dcb0cfa5be4e0291110ab0fd85c27f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:20:47.001830 11939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:46.998491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55906 (local address 127.3.211.194:44601)
0925 15:20:46.998571 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:46.998573 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:46.998581 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:20:46.998614 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:46.998615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:46.998632 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:46.998703 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:20:46.998921 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:46.999400 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:47.000670 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:47.000913 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:47.001049 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:47.001373 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:20:47.001497 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:20:47.001581 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:47.001648 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 15:20:47.001702 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 15:20:47.004279 11868 raft_consensus.cc:1261] T 2878733a76da4e3680597428f3b3b20a P 1d5976b8e2a84424b64a34b9e87adf95 [term 1 FOLLOWER]: Refusing update from remote peer 64dcb0cfa5be4e0291110ab0fd85c27f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:20:47.004393 11588 raft_consensus.cc:1261] T 2878733a76da4e3680597428f3b3b20a P d90226f631c24de2b88b8df73a780c0f [term 1 FOLLOWER]: Refusing update from remote peer 64dcb0cfa5be4e0291110ab0fd85c27f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:20:47.005002 11952 consensus_queue.cc:1035] T 2878733a76da4e3680597428f3b3b20a P 64dcb0cfa5be4e0291110ab0fd85c27f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:20:47.005184 11950 consensus_queue.cc:1035] T 2878733a76da4e3680597428f3b3b20a P 64dcb0cfa5be4e0291110ab0fd85c27f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:20:47.009305 11957 mvcc.cc:205] Tried to move back new op lower bound from 7333272154125737984 to 7333272154073899008. Current Snapshot: MvccSnapshot[applied={T|T < 7333272154125737984 or (T in {7333272154125737984})}]
I20260925 15:20:47.011786 11430 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41356:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:20:47.011914 11430 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41356:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:20:47.012881 11430 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ae6bbfe00c94422695a3080aeab48107: Sending DeleteTablet for 3 replicas of tablet 2878733a76da4e3680597428f3b3b20a
I20260925 15:20:47.013249 11708 tablet_service.cc:1514] Processing DeleteTablet for tablet 2878733a76da4e3680597428f3b3b20a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:20:47 UTC) from {username='oryx', principal='oryx/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:55894
I20260925 15:20:47.013289 11568 tablet_service.cc:1514] Processing DeleteTablet for tablet 2878733a76da4e3680597428f3b3b20a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:20:47 UTC) from {username='oryx', principal='oryx/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:53252
I20260925 15:20:47.013536 11965 tablet_replica.cc:330] T 2878733a76da4e3680597428f3b3b20a P d90226f631c24de2b88b8df73a780c0f: stopping tablet replica
I20260925 15:20:47.013511 11430 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41356:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 15:20:47.013610 11848 tablet_service.cc:1514] Processing DeleteTablet for tablet 2878733a76da4e3680597428f3b3b20a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:20:47 UTC) from {username='oryx', principal='oryx/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:40926
I20260925 15:20:47.013661 11965 raft_consensus.cc:2229] T 2878733a76da4e3680597428f3b3b20a P d90226f631c24de2b88b8df73a780c0f [term 1 FOLLOWER]: Raft consensus shutting down.
W20260925 15:20:47.013665 11430 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:20:47.013787 11965 raft_consensus.cc:2258] T 2878733a76da4e3680597428f3b3b20a P d90226f631c24de2b88b8df73a780c0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:20:47.013954 11964 tablet_replica.cc:330] T 2878733a76da4e3680597428f3b3b20a P 64dcb0cfa5be4e0291110ab0fd85c27f: stopping tablet replica
I20260925 15:20:47.014058 11964 raft_consensus.cc:2229] T 2878733a76da4e3680597428f3b3b20a P 64dcb0cfa5be4e0291110ab0fd85c27f [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:20:47.014182 11964 raft_consensus.cc:2258] T 2878733a76da4e3680597428f3b3b20a P 64dcb0cfa5be4e0291110ab0fd85c27f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:20:47.014593 11965 ts_tablet_manager.cc:1905] T 2878733a76da4e3680597428f3b3b20a P d90226f631c24de2b88b8df73a780c0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:20:47.015182 11964 ts_tablet_manager.cc:1905] T 2878733a76da4e3680597428f3b3b20a P 64dcb0cfa5be4e0291110ab0fd85c27f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:20:47.015434 11848 tablet_service.cc:1467] Processing CreateTablet for tablet f63ba2ee7604475cab58bb3c3331f56a (DEFAULT_TABLE table=test-table [id=242273b861f54819bab158b0b037af2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:20:47.015437 11708 tablet_service.cc:1467] Processing CreateTablet for tablet f63ba2ee7604475cab58bb3c3331f56a (DEFAULT_TABLE table=test-table [id=242273b861f54819bab158b0b037af2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:20:47.015551 11848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f63ba2ee7604475cab58bb3c3331f56a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:47.015551 11708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f63ba2ee7604475cab58bb3c3331f56a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:47.016446 11568 tablet_service.cc:1467] Processing CreateTablet for tablet f63ba2ee7604475cab58bb3c3331f56a (DEFAULT_TABLE table=test-table [id=242273b861f54819bab158b0b037af2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:20:47.016546 11568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f63ba2ee7604475cab58bb3c3331f56a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:47.017251 11964 ts_tablet_manager.cc:1918] T 2878733a76da4e3680597428f3b3b20a P 64dcb0cfa5be4e0291110ab0fd85c27f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:20:47.017338 11964 log.cc:1196] T 2878733a76da4e3680597428f3b3b20a P 64dcb0cfa5be4e0291110ab0fd85c27f: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-1/wal/wals/2878733a76da4e3680597428f3b3b20a
I20260925 15:20:47.017351 11966 tablet_replica.cc:330] T 2878733a76da4e3680597428f3b3b20a P 1d5976b8e2a84424b64a34b9e87adf95: stopping tablet replica
I20260925 15:20:47.017472 11966 raft_consensus.cc:2229] T 2878733a76da4e3680597428f3b3b20a P 1d5976b8e2a84424b64a34b9e87adf95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:20:47.017601 11966 raft_consensus.cc:2258] T 2878733a76da4e3680597428f3b3b20a P 1d5976b8e2a84424b64a34b9e87adf95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:20:47.017607 11964 ts_tablet_manager.cc:1939] T 2878733a76da4e3680597428f3b3b20a P 64dcb0cfa5be4e0291110ab0fd85c27f: Deleting consensus metadata
I20260925 15:20:47.017994 11944 tablet_bootstrap.cc:492] T f63ba2ee7604475cab58bb3c3331f56a P 64dcb0cfa5be4e0291110ab0fd85c27f: Bootstrap starting.
I20260925 15:20:47.018261 11416 catalog_manager.cc:4841] TS 64dcb0cfa5be4e0291110ab0fd85c27f (127.3.211.194:44601): tablet 2878733a76da4e3680597428f3b3b20a (table test-table [id=1de5f1797baa42ba8d4fabfb3e9a2d14]) successfully deleted
I20260925 15:20:47.018445 11944 tablet_bootstrap.cc:654] T f63ba2ee7604475cab58bb3c3331f56a P 64dcb0cfa5be4e0291110ab0fd85c27f: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:47.018460 11966 ts_tablet_manager.cc:1905] T 2878733a76da4e3680597428f3b3b20a P 1d5976b8e2a84424b64a34b9e87adf95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:20:47.018788 11943 tablet_bootstrap.cc:492] T f63ba2ee7604475cab58bb3c3331f56a P d90226f631c24de2b88b8df73a780c0f: Bootstrap starting.
I20260925 15:20:47.019104 11944 tablet_bootstrap.cc:492] T f63ba2ee7604475cab58bb3c3331f56a P 64dcb0cfa5be4e0291110ab0fd85c27f: No bootstrap required, opened a new log
I20260925 15:20:47.019150 11944 ts_tablet_manager.cc:1397] T f63ba2ee7604475cab58bb3c3331f56a P 64dcb0cfa5be4e0291110ab0fd85c27f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:47.019198 11943 tablet_bootstrap.cc:654] T f63ba2ee7604475cab58bb3c3331f56a P d90226f631c24de2b88b8df73a780c0f: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:47.019310 11944 raft_consensus.cc:348] T f63ba2ee7604475cab58bb3c3331f56a P 64dcb0cfa5be4e0291110ab0fd85c27f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } }
I20260925 15:20:47.019387 11944 raft_consensus.cc:374] T f63ba2ee7604475cab58bb3c3331f56a P 64dcb0cfa5be4e0291110ab0fd85c27f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:47.019409 11944 raft_consensus.cc:729] T f63ba2ee7604475cab58bb3c3331f56a P 64dcb0cfa5be4e0291110ab0fd85c27f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64dcb0cfa5be4e0291110ab0fd85c27f, State: Initialized, Role: FOLLOWER
I20260925 15:20:47.019465 11944 consensus_queue.cc:260] T f63ba2ee7604475cab58bb3c3331f56a P 64dcb0cfa5be4e0291110ab0fd85c27f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } }
I20260925 15:20:47.019551 11944 ts_tablet_manager.cc:1428] T f63ba2ee7604475cab58bb3c3331f56a P 64dcb0cfa5be4e0291110ab0fd85c27f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 15:20:47.020161 11966 ts_tablet_manager.cc:1918] T 2878733a76da4e3680597428f3b3b20a P 1d5976b8e2a84424b64a34b9e87adf95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:20:47.020241 11966 log.cc:1196] T 2878733a76da4e3680597428f3b3b20a P 1d5976b8e2a84424b64a34b9e87adf95: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-2/wal/wals/2878733a76da4e3680597428f3b3b20a
I20260925 15:20:47.020524 11942 tablet_bootstrap.cc:492] T f63ba2ee7604475cab58bb3c3331f56a P 1d5976b8e2a84424b64a34b9e87adf95: Bootstrap starting.
I20260925 15:20:47.020891 11943 tablet_bootstrap.cc:492] T f63ba2ee7604475cab58bb3c3331f56a P d90226f631c24de2b88b8df73a780c0f: No bootstrap required, opened a new log
I20260925 15:20:47.020952 11943 ts_tablet_manager.cc:1397] T f63ba2ee7604475cab58bb3c3331f56a P d90226f631c24de2b88b8df73a780c0f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:20:47.021018 11942 tablet_bootstrap.cc:654] T f63ba2ee7604475cab58bb3c3331f56a P 1d5976b8e2a84424b64a34b9e87adf95: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:47.021124 11943 raft_consensus.cc:348] T f63ba2ee7604475cab58bb3c3331f56a P d90226f631c24de2b88b8df73a780c0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } }
I20260925 15:20:47.021207 11943 raft_consensus.cc:374] T f63ba2ee7604475cab58bb3c3331f56a P d90226f631c24de2b88b8df73a780c0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:47.021222 11943 raft_consensus.cc:729] T f63ba2ee7604475cab58bb3c3331f56a P d90226f631c24de2b88b8df73a780c0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d90226f631c24de2b88b8df73a780c0f, State: Initialized, Role: FOLLOWER
I20260925 15:20:47.021276 11943 consensus_queue.cc:260] T f63ba2ee7604475cab58bb3c3331f56a P d90226f631c24de2b88b8df73a780c0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } }
I20260925 15:20:47.020538 11966 ts_tablet_manager.cc:1939] T 2878733a76da4e3680597428f3b3b20a P 1d5976b8e2a84424b64a34b9e87adf95: Deleting consensus metadata
I20260925 15:20:47.021384 11943 ts_tablet_manager.cc:1428] T f63ba2ee7604475cab58bb3c3331f56a P d90226f631c24de2b88b8df73a780c0f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:47.021687 11942 tablet_bootstrap.cc:492] T f63ba2ee7604475cab58bb3c3331f56a P 1d5976b8e2a84424b64a34b9e87adf95: No bootstrap required, opened a new log
I20260925 15:20:47.021739 11942 ts_tablet_manager.cc:1397] T f63ba2ee7604475cab58bb3c3331f56a P 1d5976b8e2a84424b64a34b9e87adf95: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:47.021878 11942 raft_consensus.cc:348] T f63ba2ee7604475cab58bb3c3331f56a P 1d5976b8e2a84424b64a34b9e87adf95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } }
I20260925 15:20:47.021948 11942 raft_consensus.cc:374] T f63ba2ee7604475cab58bb3c3331f56a P 1d5976b8e2a84424b64a34b9e87adf95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:47.021968 11942 raft_consensus.cc:729] T f63ba2ee7604475cab58bb3c3331f56a P 1d5976b8e2a84424b64a34b9e87adf95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d5976b8e2a84424b64a34b9e87adf95, State: Initialized, Role: FOLLOWER
I20260925 15:20:47.021987 11415 catalog_manager.cc:4841] TS 1d5976b8e2a84424b64a34b9e87adf95 (127.3.211.195:41405): tablet 2878733a76da4e3680597428f3b3b20a (table test-table [id=1de5f1797baa42ba8d4fabfb3e9a2d14]) successfully deleted
I20260925 15:20:47.022018 11942 consensus_queue.cc:260] T f63ba2ee7604475cab58bb3c3331f56a P 1d5976b8e2a84424b64a34b9e87adf95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } }
I20260925 15:20:47.022102 11942 ts_tablet_manager.cc:1428] T f63ba2ee7604475cab58bb3c3331f56a P 1d5976b8e2a84424b64a34b9e87adf95: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:47.023717 11965 ts_tablet_manager.cc:1918] T 2878733a76da4e3680597428f3b3b20a P d90226f631c24de2b88b8df73a780c0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:20:47.023808 11965 log.cc:1196] T 2878733a76da4e3680597428f3b3b20a P d90226f631c24de2b88b8df73a780c0f: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-0/wal/wals/2878733a76da4e3680597428f3b3b20a
I20260925 15:20:47.024083 11965 ts_tablet_manager.cc:1939] T 2878733a76da4e3680597428f3b3b20a P d90226f631c24de2b88b8df73a780c0f: Deleting consensus metadata
I20260925 15:20:47.024575 11415 catalog_manager.cc:4841] TS d90226f631c24de2b88b8df73a780c0f (127.3.211.193:38069): tablet 2878733a76da4e3680597428f3b3b20a (table test-table [id=1de5f1797baa42ba8d4fabfb3e9a2d14]) successfully deleted
W20260925 15:20:47.069957 11635 tablet.cc:2369] T f63ba2ee7604475cab58bb3c3331f56a P d90226f631c24de2b88b8df73a780c0f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:20:47.153843 11915 tablet.cc:2369] T f63ba2ee7604475cab58bb3c3331f56a P 1d5976b8e2a84424b64a34b9e87adf95: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:20:47.173615 11949 raft_consensus.cc:482] T f63ba2ee7604475cab58bb3c3331f56a P d90226f631c24de2b88b8df73a780c0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:20:47.173803 11949 raft_consensus.cc:504] T f63ba2ee7604475cab58bb3c3331f56a P d90226f631c24de2b88b8df73a780c0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } }
I20260925 15:20:47.174399 11949 leader_election.cc:288] T f63ba2ee7604475cab58bb3c3331f56a P d90226f631c24de2b88b8df73a780c0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64dcb0cfa5be4e0291110ab0fd85c27f (127.3.211.194:44601), 1d5976b8e2a84424b64a34b9e87adf95 (127.3.211.195:41405)
I20260925 15:20:47.177950 11939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:47.174920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:50813 (local address 127.3.211.194:44601)
0925 15:20:47.174977 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:47.174979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:47.175048 (+ 69us) server_negotiation.cc:407] Connection header received
0925 15:20:47.175101 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:47.175102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:47.175135 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:47.175194 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:20:47.175440 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:47.175973 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:47.177199 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:47.177571 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:47.177667 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:47.177676 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:47.177833 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 15:20:47.177877 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 15:20:47.177940 11941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:47.174825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:40757 (local address 127.3.211.195:41405)
0925 15:20:47.174882 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:47.174885 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:47.175053 (+ 168us) server_negotiation.cc:407] Connection header received
0925 15:20:47.175101 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:47.175102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:47.175113 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:47.175194 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:20:47.175432 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:47.176042 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:47.177154 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:47.177539 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:47.177636 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:47.177646 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:47.177808 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 15:20:47.177850 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 15:20:47.177997 11971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:47.174707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.194:44601 (local address 127.3.211.193:50813)
0925 15:20:47.174958 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:47.174969 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:47.175045 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:47.175210 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:47.175211 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:47.175213 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:20:47.175328 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:47.175331 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:47.176073 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:47.176075 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:47.177089 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:47.177092 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:47.177660 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:47.177661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:47.177738 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:47.177742 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:20:47.177824 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 15:20:47.177871 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":27,"spinlock_wait_cycles":768,"thread_start_us":75,"threads_started":1}
I20260925 15:20:47.177997 11970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:47.174707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.195:41405 (local address 127.3.211.193:40757)
0925 15:20:47.174956 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:47.174969 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:47.175045 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:47.175210 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:47.175211 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:47.175215 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:20:47.175328 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:47.175331 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:47.176147 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:47.176149 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:47.177040 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:47.177043 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:47.177636 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:47.177637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:47.177724 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:47.177728 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:20:47.177799 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 15:20:47.177848 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":6,"thread_start_us":77,"threads_started":1}
I20260925 15:20:47.178239 11728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63ba2ee7604475cab58bb3c3331f56a" candidate_uuid: "d90226f631c24de2b88b8df73a780c0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" is_pre_election: true
I20260925 15:20:47.178411 11728 raft_consensus.cc:2454] T f63ba2ee7604475cab58bb3c3331f56a P 64dcb0cfa5be4e0291110ab0fd85c27f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d90226f631c24de2b88b8df73a780c0f in term 0.
I20260925 15:20:47.178505 11868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63ba2ee7604475cab58bb3c3331f56a" candidate_uuid: "d90226f631c24de2b88b8df73a780c0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d5976b8e2a84424b64a34b9e87adf95" is_pre_election: true
I20260925 15:20:47.178598 11868 raft_consensus.cc:2454] T f63ba2ee7604475cab58bb3c3331f56a P 1d5976b8e2a84424b64a34b9e87adf95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d90226f631c24de2b88b8df73a780c0f in term 0.
I20260925 15:20:47.178639 11520 leader_election.cc:302] T f63ba2ee7604475cab58bb3c3331f56a P d90226f631c24de2b88b8df73a780c0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64dcb0cfa5be4e0291110ab0fd85c27f, d90226f631c24de2b88b8df73a780c0f; no voters:
I20260925 15:20:47.178812 11949 raft_consensus.cc:2781] T f63ba2ee7604475cab58bb3c3331f56a P d90226f631c24de2b88b8df73a780c0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:20:47.178889 11949 raft_consensus.cc:482] T f63ba2ee7604475cab58bb3c3331f56a P d90226f631c24de2b88b8df73a780c0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:20:47.178918 11949 raft_consensus.cc:3037] T f63ba2ee7604475cab58bb3c3331f56a P d90226f631c24de2b88b8df73a780c0f [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:47.179630 11949 raft_consensus.cc:504] T f63ba2ee7604475cab58bb3c3331f56a P d90226f631c24de2b88b8df73a780c0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } }
I20260925 15:20:47.179821 11949 leader_election.cc:288] T f63ba2ee7604475cab58bb3c3331f56a P d90226f631c24de2b88b8df73a780c0f [CANDIDATE]: Term 1 election: Requested vote from peers 64dcb0cfa5be4e0291110ab0fd85c27f (127.3.211.194:44601), 1d5976b8e2a84424b64a34b9e87adf95 (127.3.211.195:41405)
I20260925 15:20:47.179986 11868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63ba2ee7604475cab58bb3c3331f56a" candidate_uuid: "d90226f631c24de2b88b8df73a780c0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d5976b8e2a84424b64a34b9e87adf95"
I20260925 15:20:47.180091 11868 raft_consensus.cc:3037] T f63ba2ee7604475cab58bb3c3331f56a P 1d5976b8e2a84424b64a34b9e87adf95 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:47.180261 11728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f63ba2ee7604475cab58bb3c3331f56a" candidate_uuid: "d90226f631c24de2b88b8df73a780c0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f"
I20260925 15:20:47.180572 11728 raft_consensus.cc:3037] T f63ba2ee7604475cab58bb3c3331f56a P 64dcb0cfa5be4e0291110ab0fd85c27f [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:47.180917 11868 raft_consensus.cc:2454] T f63ba2ee7604475cab58bb3c3331f56a P 1d5976b8e2a84424b64a34b9e87adf95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d90226f631c24de2b88b8df73a780c0f in term 1.
I20260925 15:20:47.181126 11519 leader_election.cc:302] T f63ba2ee7604475cab58bb3c3331f56a P d90226f631c24de2b88b8df73a780c0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d5976b8e2a84424b64a34b9e87adf95, d90226f631c24de2b88b8df73a780c0f; no voters:
I20260925 15:20:47.181306 11949 raft_consensus.cc:2781] T f63ba2ee7604475cab58bb3c3331f56a P d90226f631c24de2b88b8df73a780c0f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:20:47.181460 11949 raft_consensus.cc:686] T f63ba2ee7604475cab58bb3c3331f56a P d90226f631c24de2b88b8df73a780c0f [term 1 LEADER]: Becoming Leader. State: Replica: d90226f631c24de2b88b8df73a780c0f, State: Running, Role: LEADER
I20260925 15:20:47.181545 11949 consensus_queue.cc:237] T f63ba2ee7604475cab58bb3c3331f56a P d90226f631c24de2b88b8df73a780c0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } }
I20260925 15:20:47.181663 11728 raft_consensus.cc:2454] T f63ba2ee7604475cab58bb3c3331f56a P 64dcb0cfa5be4e0291110ab0fd85c27f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d90226f631c24de2b88b8df73a780c0f in term 1.
I20260925 15:20:47.182420 11430 catalog_manager.cc:5495] T f63ba2ee7604475cab58bb3c3331f56a P d90226f631c24de2b88b8df73a780c0f reported cstate change: term changed from 0 to 1, leader changed from <none> to d90226f631c24de2b88b8df73a780c0f (127.3.211.193). New cstate: current_term: 1 leader_uuid: "d90226f631c24de2b88b8df73a780c0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } health_report { overall_health: UNKNOWN } } }
Sep 25 15:20:47 dist-test-slave-4bk3 krb5kdc[11386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349646, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.211.254@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM
I20260925 15:20:47.224273 11481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:47.217391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41370 (local address 127.3.211.254:44561)
0925 15:20:47.217473 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:47.217475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:47.217527 (+ 52us) server_negotiation.cc:407] Connection header received
0925 15:20:47.217586 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:47.217587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:47.217595 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:47.217647 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:47.218092 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:47.218577 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:47.219279 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:47.219453 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:47.219510 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:47.221697 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:47.221698 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:47.221698 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:47.221706 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:47.223066 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:47.223474 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:47.223475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:47.223476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:47.223529 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:47.223710 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:47.223711 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:47.223712 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:47.223823 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:47.223896 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:47.224055 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 15:20:47.224160 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 15:20:47.224375 11979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:47.217283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:44561 (local address 127.0.0.1:41370)
0925 15:20:47.217455 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:47.217465 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:47.217532 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:47.217674 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:47.217675 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:47.217961 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:47.218007 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:47.218009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:47.218669 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:47.218670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:47.219199 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:47.219200 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:47.219523 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:47.219524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:47.219563 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:47.220141 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:47.220146 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:47.221601 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:47.223169 (+ 1568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:47.223186 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:47.223189 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:47.223385 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:47.223591 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:47.223592 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:47.223593 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:47.223650 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:47.223885 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:47.223893 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:47.223937 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 15:20:47.224028 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 15:20:47.224227 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":82,"thread_start_us":71,"threads_started":1}
I20260925 15:20:47.226645 11430 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:41370:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 15:20:47.226881 11430 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:20:47.228827 11568 tablet_service.cc:1467] Processing CreateTablet for tablet e516781a11944b8dbe6888362abd59bf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d4153b9c950d43b2b53f34317ea256d0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:20:47.228827 11848 tablet_service.cc:1467] Processing CreateTablet for tablet e516781a11944b8dbe6888362abd59bf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d4153b9c950d43b2b53f34317ea256d0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:20:47.229000 11848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e516781a11944b8dbe6888362abd59bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:47.229000 11568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e516781a11944b8dbe6888362abd59bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:47.230455 11943 tablet_bootstrap.cc:492] T e516781a11944b8dbe6888362abd59bf P d90226f631c24de2b88b8df73a780c0f: Bootstrap starting.
I20260925 15:20:47.230587 11708 tablet_service.cc:1467] Processing CreateTablet for tablet e516781a11944b8dbe6888362abd59bf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d4153b9c950d43b2b53f34317ea256d0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:20:47.230751 11708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e516781a11944b8dbe6888362abd59bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:47.230942 11943 tablet_bootstrap.cc:654] T e516781a11944b8dbe6888362abd59bf P d90226f631c24de2b88b8df73a780c0f: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:47.231465 11942 tablet_bootstrap.cc:492] T e516781a11944b8dbe6888362abd59bf P 1d5976b8e2a84424b64a34b9e87adf95: Bootstrap starting.
I20260925 15:20:47.231840 11943 tablet_bootstrap.cc:492] T e516781a11944b8dbe6888362abd59bf P d90226f631c24de2b88b8df73a780c0f: No bootstrap required, opened a new log
I20260925 15:20:47.231895 11943 ts_tablet_manager.cc:1397] T e516781a11944b8dbe6888362abd59bf P d90226f631c24de2b88b8df73a780c0f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:47.231938 11942 tablet_bootstrap.cc:654] T e516781a11944b8dbe6888362abd59bf P 1d5976b8e2a84424b64a34b9e87adf95: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:47.232072 11944 tablet_bootstrap.cc:492] T e516781a11944b8dbe6888362abd59bf P 64dcb0cfa5be4e0291110ab0fd85c27f: Bootstrap starting.
I20260925 15:20:47.232072 11943 raft_consensus.cc:348] T e516781a11944b8dbe6888362abd59bf P d90226f631c24de2b88b8df73a780c0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } }
I20260925 15:20:47.232160 11943 raft_consensus.cc:374] T e516781a11944b8dbe6888362abd59bf P d90226f631c24de2b88b8df73a780c0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:47.232184 11943 raft_consensus.cc:729] T e516781a11944b8dbe6888362abd59bf P d90226f631c24de2b88b8df73a780c0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d90226f631c24de2b88b8df73a780c0f, State: Initialized, Role: FOLLOWER
I20260925 15:20:47.232232 11943 consensus_queue.cc:260] T e516781a11944b8dbe6888362abd59bf P d90226f631c24de2b88b8df73a780c0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } }
I20260925 15:20:47.232372 11943 ts_tablet_manager.cc:1428] T e516781a11944b8dbe6888362abd59bf P d90226f631c24de2b88b8df73a780c0f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:47.232450 11944 tablet_bootstrap.cc:654] T e516781a11944b8dbe6888362abd59bf P 64dcb0cfa5be4e0291110ab0fd85c27f: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:47.232539 11949 tablet_replica.cc:439] T e516781a11944b8dbe6888362abd59bf P d90226f631c24de2b88b8df73a780c0f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } } }
I20260925 15:20:47.232661 11949 tablet_replica.cc:442] T e516781a11944b8dbe6888362abd59bf P d90226f631c24de2b88b8df73a780c0f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:20:47.232765 11942 tablet_bootstrap.cc:492] T e516781a11944b8dbe6888362abd59bf P 1d5976b8e2a84424b64a34b9e87adf95: No bootstrap required, opened a new log
I20260925 15:20:47.232824 11942 ts_tablet_manager.cc:1397] T e516781a11944b8dbe6888362abd59bf P 1d5976b8e2a84424b64a34b9e87adf95: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:47.232964 11942 raft_consensus.cc:348] T e516781a11944b8dbe6888362abd59bf P 1d5976b8e2a84424b64a34b9e87adf95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } }
I20260925 15:20:47.233063 11944 tablet_bootstrap.cc:492] T e516781a11944b8dbe6888362abd59bf P 64dcb0cfa5be4e0291110ab0fd85c27f: No bootstrap required, opened a new log
I20260925 15:20:47.233067 11942 raft_consensus.cc:374] T e516781a11944b8dbe6888362abd59bf P 1d5976b8e2a84424b64a34b9e87adf95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:47.233104 11942 raft_consensus.cc:729] T e516781a11944b8dbe6888362abd59bf P 1d5976b8e2a84424b64a34b9e87adf95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d5976b8e2a84424b64a34b9e87adf95, State: Initialized, Role: FOLLOWER
I20260925 15:20:47.233119 11944 ts_tablet_manager.cc:1397] T e516781a11944b8dbe6888362abd59bf P 64dcb0cfa5be4e0291110ab0fd85c27f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 15:20:47.233165 11942 consensus_queue.cc:260] T e516781a11944b8dbe6888362abd59bf P 1d5976b8e2a84424b64a34b9e87adf95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } }
I20260925 15:20:47.233268 11942 ts_tablet_manager.cc:1428] T e516781a11944b8dbe6888362abd59bf P 1d5976b8e2a84424b64a34b9e87adf95: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:47.233242 11944 raft_consensus.cc:348] T e516781a11944b8dbe6888362abd59bf P 64dcb0cfa5be4e0291110ab0fd85c27f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } }
I20260925 15:20:47.233309 11944 raft_consensus.cc:374] T e516781a11944b8dbe6888362abd59bf P 64dcb0cfa5be4e0291110ab0fd85c27f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:47.233331 11944 raft_consensus.cc:729] T e516781a11944b8dbe6888362abd59bf P 64dcb0cfa5be4e0291110ab0fd85c27f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64dcb0cfa5be4e0291110ab0fd85c27f, State: Initialized, Role: FOLLOWER
I20260925 15:20:47.233377 11944 consensus_queue.cc:260] T e516781a11944b8dbe6888362abd59bf P 64dcb0cfa5be4e0291110ab0fd85c27f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } }
I20260925 15:20:47.233335 11948 tablet_replica.cc:439] T e516781a11944b8dbe6888362abd59bf P 1d5976b8e2a84424b64a34b9e87adf95: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } } }
I20260925 15:20:47.233440 11948 tablet_replica.cc:442] T e516781a11944b8dbe6888362abd59bf P 1d5976b8e2a84424b64a34b9e87adf95: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:20:47.233479 11944 ts_tablet_manager.cc:1428] T e516781a11944b8dbe6888362abd59bf P 64dcb0cfa5be4e0291110ab0fd85c27f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:47.233582 11950 tablet_replica.cc:439] T e516781a11944b8dbe6888362abd59bf P 64dcb0cfa5be4e0291110ab0fd85c27f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } } }
I20260925 15:20:47.233664 11950 tablet_replica.cc:442] T e516781a11944b8dbe6888362abd59bf P 64dcb0cfa5be4e0291110ab0fd85c27f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:20:47.341069 11950 raft_consensus.cc:482] T e516781a11944b8dbe6888362abd59bf P 64dcb0cfa5be4e0291110ab0fd85c27f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:20:47.341171 11950 raft_consensus.cc:504] T e516781a11944b8dbe6888362abd59bf P 64dcb0cfa5be4e0291110ab0fd85c27f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } }
I20260925 15:20:47.341355 11950 leader_election.cc:288] T e516781a11944b8dbe6888362abd59bf P 64dcb0cfa5be4e0291110ab0fd85c27f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d90226f631c24de2b88b8df73a780c0f (127.3.211.193:38069), 1d5976b8e2a84424b64a34b9e87adf95 (127.3.211.195:41405)
I20260925 15:20:47.341670 11868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e516781a11944b8dbe6888362abd59bf" candidate_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d5976b8e2a84424b64a34b9e87adf95" is_pre_election: true
I20260925 15:20:47.341656 11588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e516781a11944b8dbe6888362abd59bf" candidate_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d90226f631c24de2b88b8df73a780c0f" is_pre_election: true
I20260925 15:20:47.341763 11868 raft_consensus.cc:2454] T e516781a11944b8dbe6888362abd59bf P 1d5976b8e2a84424b64a34b9e87adf95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64dcb0cfa5be4e0291110ab0fd85c27f in term 0.
I20260925 15:20:47.341763 11588 raft_consensus.cc:2454] T e516781a11944b8dbe6888362abd59bf P d90226f631c24de2b88b8df73a780c0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64dcb0cfa5be4e0291110ab0fd85c27f in term 0.
I20260925 15:20:47.341930 11659 leader_election.cc:302] T e516781a11944b8dbe6888362abd59bf P 64dcb0cfa5be4e0291110ab0fd85c27f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d5976b8e2a84424b64a34b9e87adf95, 64dcb0cfa5be4e0291110ab0fd85c27f; no voters:
I20260925 15:20:47.342042 11950 raft_consensus.cc:2781] T e516781a11944b8dbe6888362abd59bf P 64dcb0cfa5be4e0291110ab0fd85c27f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:20:47.342111 11950 raft_consensus.cc:482] T e516781a11944b8dbe6888362abd59bf P 64dcb0cfa5be4e0291110ab0fd85c27f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:20:47.342142 11950 raft_consensus.cc:3037] T e516781a11944b8dbe6888362abd59bf P 64dcb0cfa5be4e0291110ab0fd85c27f [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:47.342783 11950 raft_consensus.cc:504] T e516781a11944b8dbe6888362abd59bf P 64dcb0cfa5be4e0291110ab0fd85c27f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } }
I20260925 15:20:47.342922 11950 leader_election.cc:288] T e516781a11944b8dbe6888362abd59bf P 64dcb0cfa5be4e0291110ab0fd85c27f [CANDIDATE]: Term 1 election: Requested vote from peers d90226f631c24de2b88b8df73a780c0f (127.3.211.193:38069), 1d5976b8e2a84424b64a34b9e87adf95 (127.3.211.195:41405)
I20260925 15:20:47.343144 11868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e516781a11944b8dbe6888362abd59bf" candidate_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d5976b8e2a84424b64a34b9e87adf95"
I20260925 15:20:47.343228 11868 raft_consensus.cc:3037] T e516781a11944b8dbe6888362abd59bf P 1d5976b8e2a84424b64a34b9e87adf95 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:47.343839 11868 raft_consensus.cc:2454] T e516781a11944b8dbe6888362abd59bf P 1d5976b8e2a84424b64a34b9e87adf95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64dcb0cfa5be4e0291110ab0fd85c27f in term 1.
I20260925 15:20:47.344034 11659 leader_election.cc:302] T e516781a11944b8dbe6888362abd59bf P 64dcb0cfa5be4e0291110ab0fd85c27f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d5976b8e2a84424b64a34b9e87adf95, 64dcb0cfa5be4e0291110ab0fd85c27f; no voters:
I20260925 15:20:47.344002 11588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e516781a11944b8dbe6888362abd59bf" candidate_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d90226f631c24de2b88b8df73a780c0f"
I20260925 15:20:47.344108 11588 raft_consensus.cc:3037] T e516781a11944b8dbe6888362abd59bf P d90226f631c24de2b88b8df73a780c0f [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:47.344137 11950 raft_consensus.cc:2781] T e516781a11944b8dbe6888362abd59bf P 64dcb0cfa5be4e0291110ab0fd85c27f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:20:47.344204 11950 raft_consensus.cc:686] T e516781a11944b8dbe6888362abd59bf P 64dcb0cfa5be4e0291110ab0fd85c27f [term 1 LEADER]: Becoming Leader. State: Replica: 64dcb0cfa5be4e0291110ab0fd85c27f, State: Running, Role: LEADER
I20260925 15:20:47.344264 11950 consensus_queue.cc:237] T e516781a11944b8dbe6888362abd59bf P 64dcb0cfa5be4e0291110ab0fd85c27f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } }
I20260925 15:20:47.344555 11952 tablet_replica.cc:439] T e516781a11944b8dbe6888362abd59bf P 64dcb0cfa5be4e0291110ab0fd85c27f: TxnStatusTablet state changed. Reason: New leader 64dcb0cfa5be4e0291110ab0fd85c27f. Latest consensus state: current_term: 1 leader_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } } }
I20260925 15:20:47.344630 11952 tablet_replica.cc:442] T e516781a11944b8dbe6888362abd59bf P 64dcb0cfa5be4e0291110ab0fd85c27f: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:20:47.344832 11984 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 15:20:47.344964 11984 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 15:20:47.344883 11431 catalog_manager.cc:5495] T e516781a11944b8dbe6888362abd59bf P 64dcb0cfa5be4e0291110ab0fd85c27f reported cstate change: term changed from 0 to 1, leader changed from <none> to 64dcb0cfa5be4e0291110ab0fd85c27f (127.3.211.194). New cstate: current_term: 1 leader_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } health_report { overall_health: UNKNOWN } } }
I20260925 15:20:47.344996 11588 raft_consensus.cc:2454] T e516781a11944b8dbe6888362abd59bf P d90226f631c24de2b88b8df73a780c0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64dcb0cfa5be4e0291110ab0fd85c27f in term 1.
I20260925 15:20:47.353901 11979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:47.350910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.194:44601 (local address 127.0.0.1:55916)
0925 15:20:47.350984 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:47.350998 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:47.351047 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:47.351196 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:47.351197 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:47.351198 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:20:47.351243 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:47.351245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:47.352124 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:47.352125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:47.352874 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:47.352876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:47.353400 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:47.353401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:47.353479 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:47.353503 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:20:47.353715 (+ 212us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:20:47.353727 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 15:20:47.353782 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 15:20:47.353826 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 15:20:47.353904 11939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:47.351011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55916 (local address 127.3.211.194:44601)
0925 15:20:47.351070 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:47.351073 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:47.351083 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:20:47.351110 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:47.351111 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:47.351122 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:47.351191 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:20:47.351285 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:47.352018 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:47.352960 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:47.353263 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:47.353340 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:47.353587 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:20:47.353625 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:20:47.353679 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:47.353799 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 15:20:47.353823 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 15:20:47.355283 11868 raft_consensus.cc:1261] T e516781a11944b8dbe6888362abd59bf P 1d5976b8e2a84424b64a34b9e87adf95 [term 1 FOLLOWER]: Refusing update from remote peer 64dcb0cfa5be4e0291110ab0fd85c27f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:20:47.355373 11588 raft_consensus.cc:1261] T e516781a11944b8dbe6888362abd59bf P d90226f631c24de2b88b8df73a780c0f [term 1 FOLLOWER]: Refusing update from remote peer 64dcb0cfa5be4e0291110ab0fd85c27f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:20:47.355516 11952 consensus_queue.cc:1035] T e516781a11944b8dbe6888362abd59bf P 64dcb0cfa5be4e0291110ab0fd85c27f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:20:47.355613 11950 consensus_queue.cc:1035] T e516781a11944b8dbe6888362abd59bf P 64dcb0cfa5be4e0291110ab0fd85c27f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:20:47.356598 11949 tablet_replica.cc:439] T e516781a11944b8dbe6888362abd59bf P d90226f631c24de2b88b8df73a780c0f: TxnStatusTablet state changed. Reason: New leader 64dcb0cfa5be4e0291110ab0fd85c27f. Latest consensus state: current_term: 1 leader_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } } }
I20260925 15:20:47.356805 11949 tablet_replica.cc:442] T e516781a11944b8dbe6888362abd59bf P d90226f631c24de2b88b8df73a780c0f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:20:47.356192 11948 tablet_replica.cc:439] T e516781a11944b8dbe6888362abd59bf P 1d5976b8e2a84424b64a34b9e87adf95: TxnStatusTablet state changed. Reason: New leader 64dcb0cfa5be4e0291110ab0fd85c27f. Latest consensus state: current_term: 1 leader_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } } }
I20260925 15:20:47.357087 11948 tablet_replica.cc:442] T e516781a11944b8dbe6888362abd59bf P 1d5976b8e2a84424b64a34b9e87adf95: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:20:47.357221 11950 tablet_replica.cc:439] T e516781a11944b8dbe6888362abd59bf P 64dcb0cfa5be4e0291110ab0fd85c27f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } } }
I20260925 15:20:47.357322 11950 tablet_replica.cc:442] T e516781a11944b8dbe6888362abd59bf P 64dcb0cfa5be4e0291110ab0fd85c27f: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:20:47.357537 11955 mvcc.cc:205] Tried to move back new op lower bound from 7333272155565625344 to 7333272155522600960. Current Snapshot: MvccSnapshot[applied={T|T < 7333272155565625344 or (T in {7333272155565625344})}]
I20260925 15:20:47.357779 11952 tablet_replica.cc:439] T e516781a11944b8dbe6888362abd59bf P 64dcb0cfa5be4e0291110ab0fd85c27f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } } }
I20260925 15:20:47.357860 11952 tablet_replica.cc:442] T e516781a11944b8dbe6888362abd59bf P 64dcb0cfa5be4e0291110ab0fd85c27f: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:20:47.357883 11948 tablet_replica.cc:439] T e516781a11944b8dbe6888362abd59bf P 1d5976b8e2a84424b64a34b9e87adf95: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } } }
I20260925 15:20:47.357970 11948 tablet_replica.cc:442] T e516781a11944b8dbe6888362abd59bf P 1d5976b8e2a84424b64a34b9e87adf95: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:20:47.358327 11956 mvcc.cc:205] Tried to move back new op lower bound from 7333272155565625344 to 7333272155522600960. Current Snapshot: MvccSnapshot[applied={T|T < 7333272155565625344 or (T in {7333272155565625344})}]
I20260925 15:20:47.358517 11949 tablet_replica.cc:439] T e516781a11944b8dbe6888362abd59bf P d90226f631c24de2b88b8df73a780c0f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } } }
I20260925 15:20:47.358595 11949 tablet_replica.cc:442] T e516781a11944b8dbe6888362abd59bf P d90226f631c24de2b88b8df73a780c0f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:20:47.362666 11938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:47.359415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53254 (local address 127.3.211.193:38069)
0925 15:20:47.359461 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:47.359463 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:47.359471 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:20:47.359534 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:47.359535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:47.359557 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:47.359631 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:20:47.360052 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:47.360662 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:47.361851 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:47.362052 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:47.362118 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:47.362302 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:20:47.362415 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:20:47.362474 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:47.362543 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 15:20:47.362565 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 15:20:47.369282 11939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:47.365830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:39905 (local address 127.3.211.194:44601)
0925 15:20:47.365877 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:47.365879 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:47.365890 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:20:47.365954 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:47.365955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:47.365970 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:47.366052 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:20:47.366568 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:47.367384 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:47.368617 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:47.368808 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:47.368868 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:47.369080 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:20:47.369104 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:20:47.369134 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:47.369200 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 15:20:47.369223 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 15:20:47.369449 11991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:47.365682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.194:44601 (local address 127.3.211.193:39905)
0925 15:20:47.365828 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:47.365840 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:47.365895 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:47.366055 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:47.366056 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:47.366057 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:20:47.366336 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:47.366343 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:47.367482 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:47.367483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:47.368522 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:47.368525 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:47.368916 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:47.368917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:47.368976 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:47.368998 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:20:47.369149 (+ 151us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:20:47.369150 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 15:20:47.369198 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 15:20:47.369245 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":78,"threads_started":1}
I20260925 15:20:47.372072 11728 raft_consensus.cc:1261] T f63ba2ee7604475cab58bb3c3331f56a P 64dcb0cfa5be4e0291110ab0fd85c27f [term 1 FOLLOWER]: Refusing update from remote peer d90226f631c24de2b88b8df73a780c0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:20:47.372133 11868 raft_consensus.cc:1261] T f63ba2ee7604475cab58bb3c3331f56a P 1d5976b8e2a84424b64a34b9e87adf95 [term 1 FOLLOWER]: Refusing update from remote peer d90226f631c24de2b88b8df73a780c0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:20:47.372435 11972 consensus_queue.cc:1035] T f63ba2ee7604475cab58bb3c3331f56a P d90226f631c24de2b88b8df73a780c0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:20:47.372571 11949 consensus_queue.cc:1035] T f63ba2ee7604475cab58bb3c3331f56a P d90226f631c24de2b88b8df73a780c0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:20:47.381551 11938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:47.378529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:33429 (local address 127.3.211.193:38069)
0925 15:20:47.378568 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:47.378570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:47.378741 (+ 171us) server_negotiation.cc:407] Connection header received
0925 15:20:47.378830 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:47.378831 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:47.378845 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:47.378897 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:20:47.379024 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:47.379492 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:47.380643 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:47.380974 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:47.381032 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:47.381292 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:20:47.381324 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:20:47.381382 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:47.381466 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 15:20:47.381488 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 15:20:47.381615 11997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:47.378470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:38069 (local address 127.3.211.194:33429)
0925 15:20:47.378685 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:47.378696 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:47.378769 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:47.378909 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:47.378910 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:47.378912 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:20:47.378966 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:47.378968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:47.379583 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:47.379583 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:47.380525 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:47.380529 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:47.381061 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:47.381062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:47.381123 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:47.381146 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:20:47.381406 (+ 260us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:20:47.381408 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 15:20:47.381464 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 15:20:47.381514 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":78,"threads_started":1}
I20260925 15:20:47.390964 11430 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41356:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:20:47.391086 11430 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41356:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:20:47.391835 11430 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ae6bbfe00c94422695a3080aeab48107: Sending DeleteTablet for 3 replicas of tablet f63ba2ee7604475cab58bb3c3331f56a
I20260925 15:20:47.392175 11568 tablet_service.cc:1514] Processing DeleteTablet for tablet f63ba2ee7604475cab58bb3c3331f56a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:20:47 UTC) from {username='oryx', principal='oryx/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:53252
I20260925 15:20:47.392266 11965 tablet_replica.cc:330] T f63ba2ee7604475cab58bb3c3331f56a P d90226f631c24de2b88b8df73a780c0f: stopping tablet replica
I20260925 15:20:47.392304 11965 raft_consensus.cc:2229] T f63ba2ee7604475cab58bb3c3331f56a P d90226f631c24de2b88b8df73a780c0f [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:20:47.392448 11965 raft_consensus.cc:2258] T f63ba2ee7604475cab58bb3c3331f56a P d90226f631c24de2b88b8df73a780c0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:20:47.394153 11965 ts_tablet_manager.cc:1905] T f63ba2ee7604475cab58bb3c3331f56a P d90226f631c24de2b88b8df73a780c0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:20:47.395854 11965 ts_tablet_manager.cc:1918] T f63ba2ee7604475cab58bb3c3331f56a P d90226f631c24de2b88b8df73a780c0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:20:47.395911 11965 log.cc:1196] T f63ba2ee7604475cab58bb3c3331f56a P d90226f631c24de2b88b8df73a780c0f: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-0/wal/wals/f63ba2ee7604475cab58bb3c3331f56a
I20260925 15:20:47.396127 11965 ts_tablet_manager.cc:1939] T f63ba2ee7604475cab58bb3c3331f56a P d90226f631c24de2b88b8df73a780c0f: Deleting consensus metadata
I20260925 15:20:47.396523 11848 tablet_service.cc:1514] Processing DeleteTablet for tablet f63ba2ee7604475cab58bb3c3331f56a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:20:47 UTC) from {username='oryx', principal='oryx/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:40926
I20260925 15:20:47.396642 11966 tablet_replica.cc:330] T f63ba2ee7604475cab58bb3c3331f56a P 1d5976b8e2a84424b64a34b9e87adf95: stopping tablet replica
I20260925 15:20:47.396692 11966 raft_consensus.cc:2229] T f63ba2ee7604475cab58bb3c3331f56a P 1d5976b8e2a84424b64a34b9e87adf95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:20:47.396675 11415 catalog_manager.cc:4841] TS d90226f631c24de2b88b8df73a780c0f (127.3.211.193:38069): tablet f63ba2ee7604475cab58bb3c3331f56a (table test-table [id=242273b861f54819bab158b0b037af2e]) successfully deleted
I20260925 15:20:47.396736 11966 raft_consensus.cc:2258] T f63ba2ee7604475cab58bb3c3331f56a P 1d5976b8e2a84424b64a34b9e87adf95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:20:47.397006 11708 tablet_service.cc:1514] Processing DeleteTablet for tablet f63ba2ee7604475cab58bb3c3331f56a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:20:47 UTC) from {username='oryx', principal='oryx/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:55894
I20260925 15:20:47.397102 11964 tablet_replica.cc:330] T f63ba2ee7604475cab58bb3c3331f56a P 64dcb0cfa5be4e0291110ab0fd85c27f: stopping tablet replica
I20260925 15:20:47.397115 11966 ts_tablet_manager.cc:1905] T f63ba2ee7604475cab58bb3c3331f56a P 1d5976b8e2a84424b64a34b9e87adf95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:20:47.397154 11964 raft_consensus.cc:2229] T f63ba2ee7604475cab58bb3c3331f56a P 64dcb0cfa5be4e0291110ab0fd85c27f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:20:47.397190 11964 raft_consensus.cc:2258] T f63ba2ee7604475cab58bb3c3331f56a P 64dcb0cfa5be4e0291110ab0fd85c27f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:20:47.397529 11964 ts_tablet_manager.cc:1905] T f63ba2ee7604475cab58bb3c3331f56a P 64dcb0cfa5be4e0291110ab0fd85c27f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:20:47.398679 11966 ts_tablet_manager.cc:1918] T f63ba2ee7604475cab58bb3c3331f56a P 1d5976b8e2a84424b64a34b9e87adf95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:20:47.398734 11966 log.cc:1196] T f63ba2ee7604475cab58bb3c3331f56a P 1d5976b8e2a84424b64a34b9e87adf95: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-2/wal/wals/f63ba2ee7604475cab58bb3c3331f56a
I20260925 15:20:47.398984 11966 ts_tablet_manager.cc:1939] T f63ba2ee7604475cab58bb3c3331f56a P 1d5976b8e2a84424b64a34b9e87adf95: Deleting consensus metadata
I20260925 15:20:47.399113 11964 ts_tablet_manager.cc:1918] T f63ba2ee7604475cab58bb3c3331f56a P 64dcb0cfa5be4e0291110ab0fd85c27f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:20:47.399168 11964 log.cc:1196] T f63ba2ee7604475cab58bb3c3331f56a P 64dcb0cfa5be4e0291110ab0fd85c27f: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-1/wal/wals/f63ba2ee7604475cab58bb3c3331f56a
I20260925 15:20:47.399397 11964 ts_tablet_manager.cc:1939] T f63ba2ee7604475cab58bb3c3331f56a P 64dcb0cfa5be4e0291110ab0fd85c27f: Deleting consensus metadata
I20260925 15:20:47.400022 11415 catalog_manager.cc:4841] TS 1d5976b8e2a84424b64a34b9e87adf95 (127.3.211.195:41405): tablet f63ba2ee7604475cab58bb3c3331f56a (table test-table [id=242273b861f54819bab158b0b037af2e]) successfully deleted
I20260925 15:20:47.400100 11416 catalog_manager.cc:4841] TS 64dcb0cfa5be4e0291110ab0fd85c27f (127.3.211.194:44601): tablet f63ba2ee7604475cab58bb3c3331f56a (table test-table [id=242273b861f54819bab158b0b037af2e]) successfully deleted
Sep 25 15:20:47 dist-test-slave-4bk3 krb5kdc[11386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 15:20:47 dist-test-slave-4bk3 krb5kdc[11386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:47 dist-test-slave-4bk3 krb5kdc[11386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 15:20:47.430007 11481 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:47.421171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41376 (local address 127.3.211.254:44561)
0925 15:20:47.421221 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:47.421223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:47.421497 (+ 274us) server_negotiation.cc:407] Connection header received
0925 15:20:47.421614 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:47.421615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:47.421627 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:47.421658 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:47.422481 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:47.423271 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:47.424774 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:47.425248 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:47.425314 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:47.429931 (+ 4617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41376: BlockingRecv error: recv got EOF from 127.0.0.1:41376 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 25 15:20:47 dist-test-slave-4bk3 krb5kdc[11386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM
I20260925 15:20:47.440922 11481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:47.431511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41384 (local address 127.3.211.254:44561)
0925 15:20:47.431548 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:47.431550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:47.431600 (+ 50us) server_negotiation.cc:407] Connection header received
0925 15:20:47.431674 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:47.431675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:47.431685 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:47.431758 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:47.432444 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:47.433121 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:47.434413 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:47.434705 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:47.434799 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:47.437776 (+ 2977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:47.437778 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:47.437778 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:47.437787 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:47.439561 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:47.440000 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:47.440001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:47.440002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:47.440064 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:47.440323 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:47.440324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:47.440325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:47.440485 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:47.440587 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:47.440742 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 15:20:47.440830 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 15:20:47.442708 11430 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41384:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:20:47.442888 11430 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:20:47.444433 11568 tablet_service.cc:1467] Processing CreateTablet for tablet b680496cb88c40c3be0c2552f2771a5e (DEFAULT_TABLE table=test-table [id=5aa720422f7c4371a4e148da35908b9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:20:47.444455 11708 tablet_service.cc:1467] Processing CreateTablet for tablet b680496cb88c40c3be0c2552f2771a5e (DEFAULT_TABLE table=test-table [id=5aa720422f7c4371a4e148da35908b9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:20:47.444581 11568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b680496cb88c40c3be0c2552f2771a5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:47.444587 11708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b680496cb88c40c3be0c2552f2771a5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:47.445927 11943 tablet_bootstrap.cc:492] T b680496cb88c40c3be0c2552f2771a5e P d90226f631c24de2b88b8df73a780c0f: Bootstrap starting.
I20260925 15:20:47.445982 11848 tablet_service.cc:1467] Processing CreateTablet for tablet b680496cb88c40c3be0c2552f2771a5e (DEFAULT_TABLE table=test-table [id=5aa720422f7c4371a4e148da35908b9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:20:47.446098 11848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b680496cb88c40c3be0c2552f2771a5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:47.446336 11943 tablet_bootstrap.cc:654] T b680496cb88c40c3be0c2552f2771a5e P d90226f631c24de2b88b8df73a780c0f: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:47.447403 11944 tablet_bootstrap.cc:492] T b680496cb88c40c3be0c2552f2771a5e P 64dcb0cfa5be4e0291110ab0fd85c27f: Bootstrap starting.
I20260925 15:20:47.447480 11943 tablet_bootstrap.cc:492] T b680496cb88c40c3be0c2552f2771a5e P d90226f631c24de2b88b8df73a780c0f: No bootstrap required, opened a new log
I20260925 15:20:47.447523 11943 ts_tablet_manager.cc:1397] T b680496cb88c40c3be0c2552f2771a5e P d90226f631c24de2b88b8df73a780c0f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:20:47.447669 11943 raft_consensus.cc:348] T b680496cb88c40c3be0c2552f2771a5e P d90226f631c24de2b88b8df73a780c0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } }
I20260925 15:20:47.447708 11944 tablet_bootstrap.cc:654] T b680496cb88c40c3be0c2552f2771a5e P 64dcb0cfa5be4e0291110ab0fd85c27f: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:47.447736 11943 raft_consensus.cc:374] T b680496cb88c40c3be0c2552f2771a5e P d90226f631c24de2b88b8df73a780c0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:47.447757 11943 raft_consensus.cc:729] T b680496cb88c40c3be0c2552f2771a5e P d90226f631c24de2b88b8df73a780c0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d90226f631c24de2b88b8df73a780c0f, State: Initialized, Role: FOLLOWER
I20260925 15:20:47.447800 11943 consensus_queue.cc:260] T b680496cb88c40c3be0c2552f2771a5e P d90226f631c24de2b88b8df73a780c0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } }
I20260925 15:20:47.447885 11943 ts_tablet_manager.cc:1428] T b680496cb88c40c3be0c2552f2771a5e P d90226f631c24de2b88b8df73a780c0f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:47.448037 11942 tablet_bootstrap.cc:492] T b680496cb88c40c3be0c2552f2771a5e P 1d5976b8e2a84424b64a34b9e87adf95: Bootstrap starting.
I20260925 15:20:47.448459 11942 tablet_bootstrap.cc:654] T b680496cb88c40c3be0c2552f2771a5e P 1d5976b8e2a84424b64a34b9e87adf95: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:47.449055 11944 tablet_bootstrap.cc:492] T b680496cb88c40c3be0c2552f2771a5e P 64dcb0cfa5be4e0291110ab0fd85c27f: No bootstrap required, opened a new log
I20260925 15:20:47.449105 11944 ts_tablet_manager.cc:1397] T b680496cb88c40c3be0c2552f2771a5e P 64dcb0cfa5be4e0291110ab0fd85c27f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 15:20:47.449270 11944 raft_consensus.cc:348] T b680496cb88c40c3be0c2552f2771a5e P 64dcb0cfa5be4e0291110ab0fd85c27f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } }
I20260925 15:20:47.449347 11944 raft_consensus.cc:374] T b680496cb88c40c3be0c2552f2771a5e P 64dcb0cfa5be4e0291110ab0fd85c27f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:47.449368 11944 raft_consensus.cc:729] T b680496cb88c40c3be0c2552f2771a5e P 64dcb0cfa5be4e0291110ab0fd85c27f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64dcb0cfa5be4e0291110ab0fd85c27f, State: Initialized, Role: FOLLOWER
I20260925 15:20:47.449381 11942 tablet_bootstrap.cc:492] T b680496cb88c40c3be0c2552f2771a5e P 1d5976b8e2a84424b64a34b9e87adf95: No bootstrap required, opened a new log
I20260925 15:20:47.449424 11942 ts_tablet_manager.cc:1397] T b680496cb88c40c3be0c2552f2771a5e P 1d5976b8e2a84424b64a34b9e87adf95: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 15:20:47.449409 11944 consensus_queue.cc:260] T b680496cb88c40c3be0c2552f2771a5e P 64dcb0cfa5be4e0291110ab0fd85c27f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } }
I20260925 15:20:47.449502 11944 ts_tablet_manager.cc:1428] T b680496cb88c40c3be0c2552f2771a5e P 64dcb0cfa5be4e0291110ab0fd85c27f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:47.449566 11942 raft_consensus.cc:348] T b680496cb88c40c3be0c2552f2771a5e P 1d5976b8e2a84424b64a34b9e87adf95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } }
I20260925 15:20:47.449645 11942 raft_consensus.cc:374] T b680496cb88c40c3be0c2552f2771a5e P 1d5976b8e2a84424b64a34b9e87adf95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:47.449667 11942 raft_consensus.cc:729] T b680496cb88c40c3be0c2552f2771a5e P 1d5976b8e2a84424b64a34b9e87adf95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d5976b8e2a84424b64a34b9e87adf95, State: Initialized, Role: FOLLOWER
I20260925 15:20:47.449705 11942 consensus_queue.cc:260] T b680496cb88c40c3be0c2552f2771a5e P 1d5976b8e2a84424b64a34b9e87adf95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } }
I20260925 15:20:47.449783 11942 ts_tablet_manager.cc:1428] T b680496cb88c40c3be0c2552f2771a5e P 1d5976b8e2a84424b64a34b9e87adf95: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 15:20:47.582073 11948 raft_consensus.cc:482] T b680496cb88c40c3be0c2552f2771a5e P 1d5976b8e2a84424b64a34b9e87adf95 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:20:47.582190 11948 raft_consensus.cc:504] T b680496cb88c40c3be0c2552f2771a5e P 1d5976b8e2a84424b64a34b9e87adf95 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } }
I20260925 15:20:47.582608 11948 leader_election.cc:288] T b680496cb88c40c3be0c2552f2771a5e P 1d5976b8e2a84424b64a34b9e87adf95 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d90226f631c24de2b88b8df73a780c0f (127.3.211.193:38069), 64dcb0cfa5be4e0291110ab0fd85c27f (127.3.211.194:44601)
I20260925 15:20:47.586303 11939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:47.582906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:52759 (local address 127.3.211.194:44601)
0925 15:20:47.582960 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:47.582963 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:47.583139 (+ 176us) server_negotiation.cc:407] Connection header received
0925 15:20:47.583180 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:47.583181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:47.583194 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:47.583261 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:20:47.583519 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:47.584208 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:47.585355 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:47.585821 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:47.585886 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:47.585894 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:47.586167 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 15:20:47.586216 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 15:20:47.586495 12016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:47.582836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.194:44601 (local address 127.3.211.195:52759)
0925 15:20:47.583003 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:47.583016 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:47.583093 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:47.583344 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:47.583345 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:47.583348 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:20:47.583435 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:47.583438 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:47.584294 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:47.584295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:47.585257 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:47.585259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:47.585939 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:47.585941 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:47.586016 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:47.586021 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:20:47.586306 (+ 285us) client_negotiation.cc:240] Negotiation successful
0925 15:20:47.586379 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":96,"threads_started":1}
I20260925 15:20:47.586753 11938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:47.582944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:57283 (local address 127.3.211.193:38069)
0925 15:20:47.583006 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:47.583009 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:47.583327 (+ 318us) server_negotiation.cc:407] Connection header received
0925 15:20:47.583356 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:47.583357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:47.583366 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:47.583435 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:20:47.583975 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:47.584587 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:47.585668 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:47.586179 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:47.586264 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:47.586272 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:47.586636 (+ 364us) server_negotiation.cc:299] Negotiation successful
0925 15:20:47.586682 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 15:20:47.586817 11728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b680496cb88c40c3be0c2552f2771a5e" candidate_uuid: "1d5976b8e2a84424b64a34b9e87adf95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" is_pre_election: true
I20260925 15:20:47.586956 11728 raft_consensus.cc:2454] T b680496cb88c40c3be0c2552f2771a5e P 64dcb0cfa5be4e0291110ab0fd85c27f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d5976b8e2a84424b64a34b9e87adf95 in term 0.
I20260925 15:20:47.587038 12015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:47.582841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:38069 (local address 127.3.211.195:57283)
0925 15:20:47.583166 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:47.583178 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:47.583252 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:47.583448 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:47.583449 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:47.583451 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:20:47.583872 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:47.583875 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:47.584704 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:47.584705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:47.585528 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:47.585530 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:47.586298 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:47.586300 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:47.586482 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:47.586494 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 15:20:47.586873 (+ 379us) client_negotiation.cc:240] Negotiation successful
0925 15:20:47.586934 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":103,"threads_started":1}
I20260925 15:20:47.587159 11800 leader_election.cc:302] T b680496cb88c40c3be0c2552f2771a5e P 1d5976b8e2a84424b64a34b9e87adf95 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d5976b8e2a84424b64a34b9e87adf95, 64dcb0cfa5be4e0291110ab0fd85c27f; no voters:
I20260925 15:20:47.587309 11948 raft_consensus.cc:2781] T b680496cb88c40c3be0c2552f2771a5e P 1d5976b8e2a84424b64a34b9e87adf95 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:20:47.587353 11948 raft_consensus.cc:482] T b680496cb88c40c3be0c2552f2771a5e P 1d5976b8e2a84424b64a34b9e87adf95 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:20:47.587373 11948 raft_consensus.cc:3037] T b680496cb88c40c3be0c2552f2771a5e P 1d5976b8e2a84424b64a34b9e87adf95 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:47.587316 11588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b680496cb88c40c3be0c2552f2771a5e" candidate_uuid: "1d5976b8e2a84424b64a34b9e87adf95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d90226f631c24de2b88b8df73a780c0f" is_pre_election: true
I20260925 15:20:47.587433 11588 raft_consensus.cc:2454] T b680496cb88c40c3be0c2552f2771a5e P d90226f631c24de2b88b8df73a780c0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d5976b8e2a84424b64a34b9e87adf95 in term 0.
I20260925 15:20:47.588124 11948 raft_consensus.cc:504] T b680496cb88c40c3be0c2552f2771a5e P 1d5976b8e2a84424b64a34b9e87adf95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } }
I20260925 15:20:47.588284 11948 leader_election.cc:288] T b680496cb88c40c3be0c2552f2771a5e P 1d5976b8e2a84424b64a34b9e87adf95 [CANDIDATE]: Term 1 election: Requested vote from peers d90226f631c24de2b88b8df73a780c0f (127.3.211.193:38069), 64dcb0cfa5be4e0291110ab0fd85c27f (127.3.211.194:44601)
I20260925 15:20:47.588471 11588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b680496cb88c40c3be0c2552f2771a5e" candidate_uuid: "1d5976b8e2a84424b64a34b9e87adf95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d90226f631c24de2b88b8df73a780c0f"
I20260925 15:20:47.588475 11728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b680496cb88c40c3be0c2552f2771a5e" candidate_uuid: "1d5976b8e2a84424b64a34b9e87adf95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f"
I20260925 15:20:47.588565 11728 raft_consensus.cc:3037] T b680496cb88c40c3be0c2552f2771a5e P 64dcb0cfa5be4e0291110ab0fd85c27f [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:47.588565 11588 raft_consensus.cc:3037] T b680496cb88c40c3be0c2552f2771a5e P d90226f631c24de2b88b8df73a780c0f [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:47.589362 11728 raft_consensus.cc:2454] T b680496cb88c40c3be0c2552f2771a5e P 64dcb0cfa5be4e0291110ab0fd85c27f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d5976b8e2a84424b64a34b9e87adf95 in term 1.
I20260925 15:20:47.589362 11588 raft_consensus.cc:2454] T b680496cb88c40c3be0c2552f2771a5e P d90226f631c24de2b88b8df73a780c0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d5976b8e2a84424b64a34b9e87adf95 in term 1.
I20260925 15:20:47.589618 11799 leader_election.cc:302] T b680496cb88c40c3be0c2552f2771a5e P 1d5976b8e2a84424b64a34b9e87adf95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d5976b8e2a84424b64a34b9e87adf95, d90226f631c24de2b88b8df73a780c0f; no voters:
I20260925 15:20:47.589712 11948 raft_consensus.cc:2781] T b680496cb88c40c3be0c2552f2771a5e P 1d5976b8e2a84424b64a34b9e87adf95 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:20:47.589828 11948 raft_consensus.cc:686] T b680496cb88c40c3be0c2552f2771a5e P 1d5976b8e2a84424b64a34b9e87adf95 [term 1 LEADER]: Becoming Leader. State: Replica: 1d5976b8e2a84424b64a34b9e87adf95, State: Running, Role: LEADER
I20260925 15:20:47.589941 11948 consensus_queue.cc:237] T b680496cb88c40c3be0c2552f2771a5e P 1d5976b8e2a84424b64a34b9e87adf95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } }
I20260925 15:20:47.590689 11430 catalog_manager.cc:5495] T b680496cb88c40c3be0c2552f2771a5e P 1d5976b8e2a84424b64a34b9e87adf95 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d5976b8e2a84424b64a34b9e87adf95 (127.3.211.195). New cstate: current_term: 1 leader_uuid: "1d5976b8e2a84424b64a34b9e87adf95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } health_report { overall_health: HEALTHY } } }
I20260925 15:20:47.601996 11941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:47.599093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40942 (local address 127.3.211.195:41405)
0925 15:20:47.599193 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:47.599195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:47.599206 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:20:47.599240 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:47.599241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:47.599269 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:47.599345 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:20:47.599600 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:47.600160 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:47.601093 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:47.601347 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:47.601406 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:47.601603 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:20:47.601712 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:20:47.601794 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:47.601867 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 15:20:47.601888 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 15:20:47.603104 11588 raft_consensus.cc:1261] T b680496cb88c40c3be0c2552f2771a5e P d90226f631c24de2b88b8df73a780c0f [term 1 FOLLOWER]: Refusing update from remote peer 1d5976b8e2a84424b64a34b9e87adf95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:20:47.603121 11728 raft_consensus.cc:1261] T b680496cb88c40c3be0c2552f2771a5e P 64dcb0cfa5be4e0291110ab0fd85c27f [term 1 FOLLOWER]: Refusing update from remote peer 1d5976b8e2a84424b64a34b9e87adf95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:20:47.603395 12017 consensus_queue.cc:1035] T b680496cb88c40c3be0c2552f2771a5e P 1d5976b8e2a84424b64a34b9e87adf95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:20:47.603691 11948 consensus_queue.cc:1035] T b680496cb88c40c3be0c2552f2771a5e P 1d5976b8e2a84424b64a34b9e87adf95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:20:47.607908 11430 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41384:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:20:47.608035 11430 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41384:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:20:47.608809 11430 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ae6bbfe00c94422695a3080aeab48107: Sending DeleteTablet for 3 replicas of tablet b680496cb88c40c3be0c2552f2771a5e
I20260925 15:20:47.609081 11568 tablet_service.cc:1514] Processing DeleteTablet for tablet b680496cb88c40c3be0c2552f2771a5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:20:47 UTC) from {username='oryx', principal='oryx/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:53252
I20260925 15:20:47.609117 11708 tablet_service.cc:1514] Processing DeleteTablet for tablet b680496cb88c40c3be0c2552f2771a5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:20:47 UTC) from {username='oryx', principal='oryx/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:55894
I20260925 15:20:47.609223 11964 tablet_replica.cc:330] T b680496cb88c40c3be0c2552f2771a5e P 64dcb0cfa5be4e0291110ab0fd85c27f: stopping tablet replica
I20260925 15:20:47.609234 11965 tablet_replica.cc:330] T b680496cb88c40c3be0c2552f2771a5e P d90226f631c24de2b88b8df73a780c0f: stopping tablet replica
I20260925 15:20:47.609290 11964 raft_consensus.cc:2229] T b680496cb88c40c3be0c2552f2771a5e P 64dcb0cfa5be4e0291110ab0fd85c27f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:20:47.609292 11965 raft_consensus.cc:2229] T b680496cb88c40c3be0c2552f2771a5e P d90226f631c24de2b88b8df73a780c0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:20:47.609335 11964 raft_consensus.cc:2258] T b680496cb88c40c3be0c2552f2771a5e P 64dcb0cfa5be4e0291110ab0fd85c27f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:20:47.609355 11965 raft_consensus.cc:2258] T b680496cb88c40c3be0c2552f2771a5e P d90226f631c24de2b88b8df73a780c0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:20:47.609433 11848 tablet_service.cc:1514] Processing DeleteTablet for tablet b680496cb88c40c3be0c2552f2771a5e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:20:47 UTC) from {username='oryx', principal='oryx/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:40926
I20260925 15:20:47.609589 11966 tablet_replica.cc:330] T b680496cb88c40c3be0c2552f2771a5e P 1d5976b8e2a84424b64a34b9e87adf95: stopping tablet replica
I20260925 15:20:47.609625 11964 ts_tablet_manager.cc:1905] T b680496cb88c40c3be0c2552f2771a5e P 64dcb0cfa5be4e0291110ab0fd85c27f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:20:47.609647 11966 raft_consensus.cc:2229] T b680496cb88c40c3be0c2552f2771a5e P 1d5976b8e2a84424b64a34b9e87adf95 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:20:47.609828 11966 raft_consensus.cc:2258] T b680496cb88c40c3be0c2552f2771a5e P 1d5976b8e2a84424b64a34b9e87adf95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:20:47.610118 11965 ts_tablet_manager.cc:1905] T b680496cb88c40c3be0c2552f2771a5e P d90226f631c24de2b88b8df73a780c0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:20:47.611217 11964 ts_tablet_manager.cc:1918] T b680496cb88c40c3be0c2552f2771a5e P 64dcb0cfa5be4e0291110ab0fd85c27f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:20:47.611279 11964 log.cc:1196] T b680496cb88c40c3be0c2552f2771a5e P 64dcb0cfa5be4e0291110ab0fd85c27f: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-1/wal/wals/b680496cb88c40c3be0c2552f2771a5e
I20260925 15:20:47.611699 11965 ts_tablet_manager.cc:1918] T b680496cb88c40c3be0c2552f2771a5e P d90226f631c24de2b88b8df73a780c0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:20:47.611755 11965 log.cc:1196] T b680496cb88c40c3be0c2552f2771a5e P d90226f631c24de2b88b8df73a780c0f: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-0/wal/wals/b680496cb88c40c3be0c2552f2771a5e
I20260925 15:20:47.611840 11964 ts_tablet_manager.cc:1939] T b680496cb88c40c3be0c2552f2771a5e P 64dcb0cfa5be4e0291110ab0fd85c27f: Deleting consensus metadata
I20260925 15:20:47.612035 11965 ts_tablet_manager.cc:1939] T b680496cb88c40c3be0c2552f2771a5e P d90226f631c24de2b88b8df73a780c0f: Deleting consensus metadata
I20260925 15:20:47.612448 11416 catalog_manager.cc:4841] TS 64dcb0cfa5be4e0291110ab0fd85c27f (127.3.211.194:44601): tablet b680496cb88c40c3be0c2552f2771a5e (table test-table [id=5aa720422f7c4371a4e148da35908b9c]) successfully deleted
I20260925 15:20:47.612385 11429 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41384:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:20:47.612556 11429 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:20:47.612825 11966 ts_tablet_manager.cc:1905] T b680496cb88c40c3be0c2552f2771a5e P 1d5976b8e2a84424b64a34b9e87adf95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:20:47.614375 11966 ts_tablet_manager.cc:1918] T b680496cb88c40c3be0c2552f2771a5e P 1d5976b8e2a84424b64a34b9e87adf95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:20:47.614435 11966 log.cc:1196] T b680496cb88c40c3be0c2552f2771a5e P 1d5976b8e2a84424b64a34b9e87adf95: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-2/wal/wals/b680496cb88c40c3be0c2552f2771a5e
I20260925 15:20:47.614588 11966 ts_tablet_manager.cc:1939] T b680496cb88c40c3be0c2552f2771a5e P 1d5976b8e2a84424b64a34b9e87adf95: Deleting consensus metadata
I20260925 15:20:47.614863 11708 tablet_service.cc:1467] Processing CreateTablet for tablet 3afaefea92694946bd236d12b1fe6949 (DEFAULT_TABLE table=test-table [id=4b4a42bccb714f589d7f8c6ca369d70b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:20:47.614888 11415 catalog_manager.cc:4841] TS d90226f631c24de2b88b8df73a780c0f (127.3.211.193:38069): tablet b680496cb88c40c3be0c2552f2771a5e (table test-table [id=5aa720422f7c4371a4e148da35908b9c]) successfully deleted
I20260925 15:20:47.614988 11708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3afaefea92694946bd236d12b1fe6949. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:47.615079 11568 tablet_service.cc:1467] Processing CreateTablet for tablet 3afaefea92694946bd236d12b1fe6949 (DEFAULT_TABLE table=test-table [id=4b4a42bccb714f589d7f8c6ca369d70b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:20:47.615135 11848 tablet_service.cc:1467] Processing CreateTablet for tablet 3afaefea92694946bd236d12b1fe6949 (DEFAULT_TABLE table=test-table [id=4b4a42bccb714f589d7f8c6ca369d70b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:20:47.615197 11568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3afaefea92694946bd236d12b1fe6949. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:47.615221 11848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3afaefea92694946bd236d12b1fe6949. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:47.616433 11944 tablet_bootstrap.cc:492] T 3afaefea92694946bd236d12b1fe6949 P 64dcb0cfa5be4e0291110ab0fd85c27f: Bootstrap starting.
I20260925 15:20:47.616540 11942 tablet_bootstrap.cc:492] T 3afaefea92694946bd236d12b1fe6949 P 1d5976b8e2a84424b64a34b9e87adf95: Bootstrap starting.
I20260925 15:20:47.616788 11944 tablet_bootstrap.cc:654] T 3afaefea92694946bd236d12b1fe6949 P 64dcb0cfa5be4e0291110ab0fd85c27f: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:47.616942 11942 tablet_bootstrap.cc:654] T 3afaefea92694946bd236d12b1fe6949 P 1d5976b8e2a84424b64a34b9e87adf95: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:47.617060 11943 tablet_bootstrap.cc:492] T 3afaefea92694946bd236d12b1fe6949 P d90226f631c24de2b88b8df73a780c0f: Bootstrap starting.
I20260925 15:20:47.617376 11943 tablet_bootstrap.cc:654] T 3afaefea92694946bd236d12b1fe6949 P d90226f631c24de2b88b8df73a780c0f: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:47.617378 11944 tablet_bootstrap.cc:492] T 3afaefea92694946bd236d12b1fe6949 P 64dcb0cfa5be4e0291110ab0fd85c27f: No bootstrap required, opened a new log
I20260925 15:20:47.617447 11944 ts_tablet_manager.cc:1397] T 3afaefea92694946bd236d12b1fe6949 P 64dcb0cfa5be4e0291110ab0fd85c27f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 15:20:47.617555 11415 catalog_manager.cc:4841] TS 1d5976b8e2a84424b64a34b9e87adf95 (127.3.211.195:41405): tablet b680496cb88c40c3be0c2552f2771a5e (table test-table [id=5aa720422f7c4371a4e148da35908b9c]) successfully deleted
I20260925 15:20:47.617584 11944 raft_consensus.cc:348] T 3afaefea92694946bd236d12b1fe6949 P 64dcb0cfa5be4e0291110ab0fd85c27f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } }
I20260925 15:20:47.617666 11944 raft_consensus.cc:374] T 3afaefea92694946bd236d12b1fe6949 P 64dcb0cfa5be4e0291110ab0fd85c27f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:47.617712 11944 raft_consensus.cc:729] T 3afaefea92694946bd236d12b1fe6949 P 64dcb0cfa5be4e0291110ab0fd85c27f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64dcb0cfa5be4e0291110ab0fd85c27f, State: Initialized, Role: FOLLOWER
I20260925 15:20:47.617784 11944 consensus_queue.cc:260] T 3afaefea92694946bd236d12b1fe6949 P 64dcb0cfa5be4e0291110ab0fd85c27f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } }
I20260925 15:20:47.617894 11944 ts_tablet_manager.cc:1428] T 3afaefea92694946bd236d12b1fe6949 P 64dcb0cfa5be4e0291110ab0fd85c27f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:47.618079 11943 tablet_bootstrap.cc:492] T 3afaefea92694946bd236d12b1fe6949 P d90226f631c24de2b88b8df73a780c0f: No bootstrap required, opened a new log
I20260925 15:20:47.618131 11943 ts_tablet_manager.cc:1397] T 3afaefea92694946bd236d12b1fe6949 P d90226f631c24de2b88b8df73a780c0f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:47.618248 11943 raft_consensus.cc:348] T 3afaefea92694946bd236d12b1fe6949 P d90226f631c24de2b88b8df73a780c0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } }
I20260925 15:20:47.618338 11943 raft_consensus.cc:374] T 3afaefea92694946bd236d12b1fe6949 P d90226f631c24de2b88b8df73a780c0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:47.618360 11943 raft_consensus.cc:729] T 3afaefea92694946bd236d12b1fe6949 P d90226f631c24de2b88b8df73a780c0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d90226f631c24de2b88b8df73a780c0f, State: Initialized, Role: FOLLOWER
I20260925 15:20:47.618407 11943 consensus_queue.cc:260] T 3afaefea92694946bd236d12b1fe6949 P d90226f631c24de2b88b8df73a780c0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } }
I20260925 15:20:47.618505 11943 ts_tablet_manager.cc:1428] T 3afaefea92694946bd236d12b1fe6949 P d90226f631c24de2b88b8df73a780c0f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 15:20:47.618721 11942 tablet_bootstrap.cc:492] T 3afaefea92694946bd236d12b1fe6949 P 1d5976b8e2a84424b64a34b9e87adf95: No bootstrap required, opened a new log
I20260925 15:20:47.618781 11942 ts_tablet_manager.cc:1397] T 3afaefea92694946bd236d12b1fe6949 P 1d5976b8e2a84424b64a34b9e87adf95: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 15:20:47.618935 11942 raft_consensus.cc:348] T 3afaefea92694946bd236d12b1fe6949 P 1d5976b8e2a84424b64a34b9e87adf95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } }
I20260925 15:20:47.618993 11942 raft_consensus.cc:374] T 3afaefea92694946bd236d12b1fe6949 P 1d5976b8e2a84424b64a34b9e87adf95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:47.619007 11942 raft_consensus.cc:729] T 3afaefea92694946bd236d12b1fe6949 P 1d5976b8e2a84424b64a34b9e87adf95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d5976b8e2a84424b64a34b9e87adf95, State: Initialized, Role: FOLLOWER
I20260925 15:20:47.619065 11942 consensus_queue.cc:260] T 3afaefea92694946bd236d12b1fe6949 P 1d5976b8e2a84424b64a34b9e87adf95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } }
I20260925 15:20:47.619164 11942 ts_tablet_manager.cc:1428] T 3afaefea92694946bd236d12b1fe6949 P 1d5976b8e2a84424b64a34b9e87adf95: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:47.705322 11949 raft_consensus.cc:482] T 3afaefea92694946bd236d12b1fe6949 P d90226f631c24de2b88b8df73a780c0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:20:47.705449 11949 raft_consensus.cc:504] T 3afaefea92694946bd236d12b1fe6949 P d90226f631c24de2b88b8df73a780c0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } }
I20260925 15:20:47.705612 11949 leader_election.cc:288] T 3afaefea92694946bd236d12b1fe6949 P d90226f631c24de2b88b8df73a780c0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64dcb0cfa5be4e0291110ab0fd85c27f (127.3.211.194:44601), 1d5976b8e2a84424b64a34b9e87adf95 (127.3.211.195:41405)
I20260925 15:20:47.705888 11728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3afaefea92694946bd236d12b1fe6949" candidate_uuid: "d90226f631c24de2b88b8df73a780c0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" is_pre_election: true
I20260925 15:20:47.705888 11868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3afaefea92694946bd236d12b1fe6949" candidate_uuid: "d90226f631c24de2b88b8df73a780c0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d5976b8e2a84424b64a34b9e87adf95" is_pre_election: true
I20260925 15:20:47.705992 11728 raft_consensus.cc:2454] T 3afaefea92694946bd236d12b1fe6949 P 64dcb0cfa5be4e0291110ab0fd85c27f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d90226f631c24de2b88b8df73a780c0f in term 0.
I20260925 15:20:47.705991 11868 raft_consensus.cc:2454] T 3afaefea92694946bd236d12b1fe6949 P 1d5976b8e2a84424b64a34b9e87adf95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d90226f631c24de2b88b8df73a780c0f in term 0.
I20260925 15:20:47.706151 11519 leader_election.cc:302] T 3afaefea92694946bd236d12b1fe6949 P d90226f631c24de2b88b8df73a780c0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d5976b8e2a84424b64a34b9e87adf95, d90226f631c24de2b88b8df73a780c0f; no voters:
I20260925 15:20:47.706239 11949 raft_consensus.cc:2781] T 3afaefea92694946bd236d12b1fe6949 P d90226f631c24de2b88b8df73a780c0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:20:47.706277 11949 raft_consensus.cc:482] T 3afaefea92694946bd236d12b1fe6949 P d90226f631c24de2b88b8df73a780c0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:20:47.706339 11949 raft_consensus.cc:3037] T 3afaefea92694946bd236d12b1fe6949 P d90226f631c24de2b88b8df73a780c0f [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:47.707106 11949 raft_consensus.cc:504] T 3afaefea92694946bd236d12b1fe6949 P d90226f631c24de2b88b8df73a780c0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } }
I20260925 15:20:47.707255 11949 leader_election.cc:288] T 3afaefea92694946bd236d12b1fe6949 P d90226f631c24de2b88b8df73a780c0f [CANDIDATE]: Term 1 election: Requested vote from peers 64dcb0cfa5be4e0291110ab0fd85c27f (127.3.211.194:44601), 1d5976b8e2a84424b64a34b9e87adf95 (127.3.211.195:41405)
I20260925 15:20:47.707401 11728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3afaefea92694946bd236d12b1fe6949" candidate_uuid: "d90226f631c24de2b88b8df73a780c0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f"
I20260925 15:20:47.707489 11728 raft_consensus.cc:3037] T 3afaefea92694946bd236d12b1fe6949 P 64dcb0cfa5be4e0291110ab0fd85c27f [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:47.707507 11868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3afaefea92694946bd236d12b1fe6949" candidate_uuid: "d90226f631c24de2b88b8df73a780c0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d5976b8e2a84424b64a34b9e87adf95"
I20260925 15:20:47.707602 11868 raft_consensus.cc:3037] T 3afaefea92694946bd236d12b1fe6949 P 1d5976b8e2a84424b64a34b9e87adf95 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:47.708287 11728 raft_consensus.cc:2454] T 3afaefea92694946bd236d12b1fe6949 P 64dcb0cfa5be4e0291110ab0fd85c27f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d90226f631c24de2b88b8df73a780c0f in term 1.
I20260925 15:20:47.708498 11520 leader_election.cc:302] T 3afaefea92694946bd236d12b1fe6949 P d90226f631c24de2b88b8df73a780c0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64dcb0cfa5be4e0291110ab0fd85c27f, d90226f631c24de2b88b8df73a780c0f; no voters:
I20260925 15:20:47.708535 11868 raft_consensus.cc:2454] T 3afaefea92694946bd236d12b1fe6949 P 1d5976b8e2a84424b64a34b9e87adf95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d90226f631c24de2b88b8df73a780c0f in term 1.
I20260925 15:20:47.708617 11949 raft_consensus.cc:2781] T 3afaefea92694946bd236d12b1fe6949 P d90226f631c24de2b88b8df73a780c0f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:20:47.708665 11949 raft_consensus.cc:686] T 3afaefea92694946bd236d12b1fe6949 P d90226f631c24de2b88b8df73a780c0f [term 1 LEADER]: Becoming Leader. State: Replica: d90226f631c24de2b88b8df73a780c0f, State: Running, Role: LEADER
I20260925 15:20:47.708719 11949 consensus_queue.cc:237] T 3afaefea92694946bd236d12b1fe6949 P d90226f631c24de2b88b8df73a780c0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } }
I20260925 15:20:47.709369 11430 catalog_manager.cc:5495] T 3afaefea92694946bd236d12b1fe6949 P d90226f631c24de2b88b8df73a780c0f reported cstate change: term changed from 0 to 1, leader changed from <none> to d90226f631c24de2b88b8df73a780c0f (127.3.211.193). New cstate: current_term: 1 leader_uuid: "d90226f631c24de2b88b8df73a780c0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d90226f631c24de2b88b8df73a780c0f" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 38069 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 } health_report { overall_health: UNKNOWN } } }
I20260925 15:20:47.739404 11938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:47.736950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53264 (local address 127.3.211.193:38069)
0925 15:20:47.736996 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:47.736998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:47.737007 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:20:47.737078 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:47.737079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:47.737091 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:47.737145 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:20:47.737259 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:47.737911 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:47.738690 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:47.738970 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:47.739044 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:47.739157 (+ 113us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:20:47.739192 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:20:47.739245 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:47.739294 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 15:20:47.739316 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 15:20:47.741724 11728 raft_consensus.cc:1261] T 3afaefea92694946bd236d12b1fe6949 P 64dcb0cfa5be4e0291110ab0fd85c27f [term 1 FOLLOWER]: Refusing update from remote peer d90226f631c24de2b88b8df73a780c0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:20:47.741915 11868 raft_consensus.cc:1261] T 3afaefea92694946bd236d12b1fe6949 P 1d5976b8e2a84424b64a34b9e87adf95 [term 1 FOLLOWER]: Refusing update from remote peer d90226f631c24de2b88b8df73a780c0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:20:47.741983 11949 consensus_queue.cc:1035] T 3afaefea92694946bd236d12b1fe6949 P d90226f631c24de2b88b8df73a780c0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "64dcb0cfa5be4e0291110ab0fd85c27f" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 44601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:20:47.742090 11972 consensus_queue.cc:1035] T 3afaefea92694946bd236d12b1fe6949 P d90226f631c24de2b88b8df73a780c0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d5976b8e2a84424b64a34b9e87adf95" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:20:47.752786 11430 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41384:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:20:47.752918 11430 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41384:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:20:47.753647 11430 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ae6bbfe00c94422695a3080aeab48107: Sending DeleteTablet for 3 replicas of tablet 3afaefea92694946bd236d12b1fe6949
I20260925 15:20:47.753904 11568 tablet_service.cc:1514] Processing DeleteTablet for tablet 3afaefea92694946bd236d12b1fe6949 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:20:47 UTC) from {username='oryx', principal='oryx/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:53252
I20260925 15:20:47.753958 11848 tablet_service.cc:1514] Processing DeleteTablet for tablet 3afaefea92694946bd236d12b1fe6949 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:20:47 UTC) from {username='oryx', principal='oryx/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:40926
I20260925 15:20:47.754009 11965 tablet_replica.cc:330] T 3afaefea92694946bd236d12b1fe6949 P d90226f631c24de2b88b8df73a780c0f: stopping tablet replica
I20260925 15:20:47.754012 11708 tablet_service.cc:1514] Processing DeleteTablet for tablet 3afaefea92694946bd236d12b1fe6949 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:20:47 UTC) from {username='oryx', principal='oryx/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:55894
I20260925 15:20:47.754056 11965 raft_consensus.cc:2229] T 3afaefea92694946bd236d12b1fe6949 P d90226f631c24de2b88b8df73a780c0f [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:20:47.754056 11966 tablet_replica.cc:330] T 3afaefea92694946bd236d12b1fe6949 P 1d5976b8e2a84424b64a34b9e87adf95: stopping tablet replica
I20260925 15:20:47.754089 11964 tablet_replica.cc:330] T 3afaefea92694946bd236d12b1fe6949 P 64dcb0cfa5be4e0291110ab0fd85c27f: stopping tablet replica
I20260925 15:20:47.754107 11966 raft_consensus.cc:2229] T 3afaefea92694946bd236d12b1fe6949 P 1d5976b8e2a84424b64a34b9e87adf95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:20:47.754117 11965 raft_consensus.cc:2258] T 3afaefea92694946bd236d12b1fe6949 P d90226f631c24de2b88b8df73a780c0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:20:47.754139 11966 raft_consensus.cc:2258] T 3afaefea92694946bd236d12b1fe6949 P 1d5976b8e2a84424b64a34b9e87adf95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:20:47.754143 11964 raft_consensus.cc:2229] T 3afaefea92694946bd236d12b1fe6949 P 64dcb0cfa5be4e0291110ab0fd85c27f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:20:47.754179 11964 raft_consensus.cc:2258] T 3afaefea92694946bd236d12b1fe6949 P 64dcb0cfa5be4e0291110ab0fd85c27f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:20:47.754506 11965 ts_tablet_manager.cc:1905] T 3afaefea92694946bd236d12b1fe6949 P d90226f631c24de2b88b8df73a780c0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:20:47.754676 11966 ts_tablet_manager.cc:1905] T 3afaefea92694946bd236d12b1fe6949 P 1d5976b8e2a84424b64a34b9e87adf95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:20:47.754680 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 11503
I20260925 15:20:47.754717 11964 ts_tablet_manager.cc:1905] T 3afaefea92694946bd236d12b1fe6949 P 64dcb0cfa5be4e0291110ab0fd85c27f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:20:47.756516 11964 ts_tablet_manager.cc:1918] T 3afaefea92694946bd236d12b1fe6949 P 64dcb0cfa5be4e0291110ab0fd85c27f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:20:47.756582 11964 log.cc:1196] T 3afaefea92694946bd236d12b1fe6949 P 64dcb0cfa5be4e0291110ab0fd85c27f: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-1/wal/wals/3afaefea92694946bd236d12b1fe6949
I20260925 15:20:47.756808 11964 ts_tablet_manager.cc:1939] T 3afaefea92694946bd236d12b1fe6949 P 64dcb0cfa5be4e0291110ab0fd85c27f: Deleting consensus metadata
I20260925 15:20:47.756289 11966 ts_tablet_manager.cc:1918] T 3afaefea92694946bd236d12b1fe6949 P 1d5976b8e2a84424b64a34b9e87adf95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:20:47.756914 11966 log.cc:1196] T 3afaefea92694946bd236d12b1fe6949 P 1d5976b8e2a84424b64a34b9e87adf95: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790349610595785-3919-0/minicluster-data/ts-2/wal/wals/3afaefea92694946bd236d12b1fe6949
I20260925 15:20:47.757144 11966 ts_tablet_manager.cc:1939] T 3afaefea92694946bd236d12b1fe6949 P 1d5976b8e2a84424b64a34b9e87adf95: Deleting consensus metadata
I20260925 15:20:47.757207 11416 catalog_manager.cc:4841] TS 64dcb0cfa5be4e0291110ab0fd85c27f (127.3.211.194:44601): tablet 3afaefea92694946bd236d12b1fe6949 (table test-table [id=4b4a42bccb714f589d7f8c6ca369d70b]) successfully deleted
I20260925 15:20:47.757469 11415 catalog_manager.cc:4841] TS 1d5976b8e2a84424b64a34b9e87adf95 (127.3.211.195:41405): tablet 3afaefea92694946bd236d12b1fe6949 (table test-table [id=4b4a42bccb714f589d7f8c6ca369d70b]) successfully deleted
W20260925 15:20:47.761618 11415 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.211.193:38069 (error 108)
I20260925 15:20:47.761850 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 11643
I20260925 15:20:47.764550 12030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:47.762033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:38069 (local address 127.0.0.1:53268)
0925 15:20:47.764447 (+ 2414us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:47.764506 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.193:38069: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2385,"thread_start_us":38,"threads_started":1}
W20260925 15:20:47.764700 11415 catalog_manager.cc:4584] TS d90226f631c24de2b88b8df73a780c0f (127.3.211.193:38069): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3afaefea92694946bd236d12b1fe6949: Network error: Client connection negotiation failed: client connection to 127.3.211.193:38069: connect: Connection refused (error 111)
I20260925 15:20:47.768692 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 11783
I20260925 15:20:47.774598 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 11399
2026-09-25T15:20:47Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2740 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349610595785-3919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:20:47 dist-test-slave-4bk3 krb5kdc[12033](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:20:47 dist-test-slave-4bk3 krb5kdc[12033](info): set up 2 sockets
Sep 25 15:20:47 dist-test-slave-4bk3 krb5kdc[12033](info): commencing operation
krb5kdc: starting...
W20260925 15:20:48.841106 3919 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Sep 25 15:20:48 dist-test-slave-4bk3 krb5kdc[12033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:20:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:20:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.211.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
I20260925 15:20:48.968072 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:38869
--webserver_interface=127.3.211.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:38575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.211.254:38869
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:20:49.051008 12046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:49.051237 12046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:49.051262 12046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:49.052412 12046 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:20:49.052449 12046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:49.052464 12046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:49.052476 12046 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:20:49.052489 12046 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:20:49.053965 12046 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:38575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349610595785-3919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.211.254:38869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:38869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.211.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:49.054167 12046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:49.054342 12046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:49.059441 12051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:49.059441 12052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:49.059469 12054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:49.059677 12046 server_base.cc:1034] running on GCE node
I20260925 15:20:49.059979 12046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:49.060298 12046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:49.061512 12046 hybrid_clock.cc:648] HybridClock initialized: now 1790349649061485 us; error 37 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T15:20:49Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1290 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:20:49 dist-test-slave-4bk3 krb5kdc[12059](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:20:49 dist-test-slave-4bk3 krb5kdc[12059](info): set up 2 sockets
Sep 25 15:20:49 dist-test-slave-4bk3 krb5kdc[12059](info): commencing operation
krb5kdc: starting...
W20260925 15:20:50.121402 3919 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Sep 25 15:20:50 dist-test-slave-4bk3 krb5kdc[12059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:20:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:20:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.211.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
I20260925 15:20:50.252614 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:38693
--webserver_interface=127.3.211.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:42007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.211.254:38693
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:20:50.339586 12072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:50.339847 12072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:50.339874 12072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:50.341482 12072 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:20:50.341555 12072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:50.341571 12072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:50.341589 12072 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:20:50.341602 12072 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:20:50.343329 12072 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:42007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.211.254:38693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:38693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.211.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:50.343680 12072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:50.343932 12072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:50.349189 12078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:50.349189 12080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:50.349188 12077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:50.350163 12072 server_base.cc:1034] running on GCE node
I20260925 15:20:50.350435 12072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:50.350757 12072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:50.351926 12072 hybrid_clock.cc:648] HybridClock initialized: now 1790349650351907 us; error 38 us; skew 500 ppm
Sep 25 15:20:50 dist-test-slave-4bk3 krb5kdc[12059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349650, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:50.363031 12072 init.cc:377] Logged in from keytab as kudu/127.3.211.254@KRBTEST.COM (short username kudu)
I20260925 15:20:50.363585 12072 webserver.cc:460] Webserver started at http://127.3.211.254:44619/ using document root <none> and password file <none>
I20260925 15:20:50.363844 12072 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:50.363914 12072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:50.364064 12072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:50.365272 12072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/master-0/data/instance:
uuid: "5b95d329f4c14b61be453c8c690c4110"
format_stamp: "Formatted at 2026-09-25 15:20:50 on dist-test-slave-4bk3"
server_key: "b9c6a85c2572166eb8fd502b281fd322"
server_key_iv: "43c3e7b9b4b8e67290bddf210e213684"
server_key_version: "encryptionkey@0"
I20260925 15:20:50.365659 12072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/master-0/wal/instance:
uuid: "5b95d329f4c14b61be453c8c690c4110"
format_stamp: "Formatted at 2026-09-25 15:20:50 on dist-test-slave-4bk3"
server_key: "b9c6a85c2572166eb8fd502b281fd322"
server_key_iv: "43c3e7b9b4b8e67290bddf210e213684"
server_key_version: "encryptionkey@0"
I20260925 15:20:50.367111 12072 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 15:20:50.368254 12087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:50.368783 12072 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:20:50.368924 12072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/master-0/wal
uuid: "5b95d329f4c14b61be453c8c690c4110"
format_stamp: "Formatted at 2026-09-25 15:20:50 on dist-test-slave-4bk3"
server_key: "b9c6a85c2572166eb8fd502b281fd322"
server_key_iv: "43c3e7b9b4b8e67290bddf210e213684"
server_key_version: "encryptionkey@0"
I20260925 15:20:50.369019 12072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:50.377544 12072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:50.380436 12072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:50.380620 12072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:50.384935 12072 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.254:38693
I20260925 15:20:50.385030 12139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.254:38693 every 8 connection(s)
I20260925 15:20:50.385386 12072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
I20260925 15:20:50.386421 12140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:20:50.388648 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 12072
I20260925 15:20:50.388888 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/master-0/wal/instance
I20260925 15:20:50.389065 3919 external_mini_cluster.cc:1428] Setting key 93ec82760f583c4492d77a010235f908
I20260925 15:20:50.389431 12140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b95d329f4c14b61be453c8c690c4110: Bootstrap starting.
I20260925 15:20:50.390228 12140 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b95d329f4c14b61be453c8c690c4110: Neither blocks nor log segments found. Creating new log.
I20260925 15:20:50.390590 12140 log.cc:824] T 00000000000000000000000000000000 P 5b95d329f4c14b61be453c8c690c4110: Log is configured to *not* fsync() on all Append() calls
I20260925 15:20:50.391459 12140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b95d329f4c14b61be453c8c690c4110: No bootstrap required, opened a new log
I20260925 15:20:50.393882 12140 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b95d329f4c14b61be453c8c690c4110 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b95d329f4c14b61be453c8c690c4110" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38693 } }
I20260925 15:20:50.394068 12140 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b95d329f4c14b61be453c8c690c4110 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:20:50.394091 12140 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b95d329f4c14b61be453c8c690c4110 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b95d329f4c14b61be453c8c690c4110, State: Initialized, Role: FOLLOWER
I20260925 15:20:50.394230 12140 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b95d329f4c14b61be453c8c690c4110 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b95d329f4c14b61be453c8c690c4110" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38693 } }
I20260925 15:20:50.394304 12140 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b95d329f4c14b61be453c8c690c4110 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:20:50.394356 12140 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b95d329f4c14b61be453c8c690c4110 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:20:50.394407 12140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b95d329f4c14b61be453c8c690c4110 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:20:50.395152 12140 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b95d329f4c14b61be453c8c690c4110 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b95d329f4c14b61be453c8c690c4110" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38693 } }
I20260925 15:20:50.395268 12140 leader_election.cc:302] T 00000000000000000000000000000000 P 5b95d329f4c14b61be453c8c690c4110 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b95d329f4c14b61be453c8c690c4110; no voters:
I20260925 15:20:50.395439 12140 leader_election.cc:288] T 00000000000000000000000000000000 P 5b95d329f4c14b61be453c8c690c4110 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:20:50.395524 12145 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b95d329f4c14b61be453c8c690c4110 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:20:50.395704 12145 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b95d329f4c14b61be453c8c690c4110 [term 1 LEADER]: Becoming Leader. State: Replica: 5b95d329f4c14b61be453c8c690c4110, State: Running, Role: LEADER
I20260925 15:20:50.395707 12140 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b95d329f4c14b61be453c8c690c4110 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:20:50.395895 12145 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b95d329f4c14b61be453c8c690c4110 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b95d329f4c14b61be453c8c690c4110" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38693 } }
I20260925 15:20:50.396320 12146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b95d329f4c14b61be453c8c690c4110 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b95d329f4c14b61be453c8c690c4110" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b95d329f4c14b61be453c8c690c4110" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38693 } } }
I20260925 15:20:50.396441 12146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b95d329f4c14b61be453c8c690c4110 [sys.catalog]: This master's current role is: LEADER
I20260925 15:20:50.396747 12147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b95d329f4c14b61be453c8c690c4110 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b95d329f4c14b61be453c8c690c4110. Latest consensus state: current_term: 1 leader_uuid: "5b95d329f4c14b61be453c8c690c4110" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b95d329f4c14b61be453c8c690c4110" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38693 } } }
I20260925 15:20:50.396888 12147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b95d329f4c14b61be453c8c690c4110 [sys.catalog]: This master's current role is: LEADER
I20260925 15:20:50.397198 12154 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:20:50.397850 12154 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 15:20:50 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:20:50.399737 12154 catalog_manager.cc:1347] Generated new cluster ID: 0b3f68beb6de4bbd973af673a50f42e4
I20260925 15:20:50.399799 12154 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:20:50.403215 12143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:50.390061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35424 (local address 127.3.211.254:38693)
0925 15:20:50.390308 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:50.390319 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:50.390360 (+ 41us) server_negotiation.cc:407] Connection header received
0925 15:20:50.391029 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:50.391037 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:50.391130 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:50.391307 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:50.392201 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:50.393791 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:50.394876 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:50.395543 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:50.395659 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:50.398975 (+ 3316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:50.398980 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:50.398980 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:50.398996 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:50.401320 (+ 2324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:50.401789 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:50.401793 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:50.401793 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:50.401913 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:50.402152 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:50.402153 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:50.402154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:50.402496 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:50.402740 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:50.402823 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 15:20:50.402974 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":107,"threads_started":1}
I20260925 15:20:50.418795 12154 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:20:50.419581 12154 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:20:50.429605 12154 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b95d329f4c14b61be453c8c690c4110: Generated new TSK 0
I20260925 15:20:50.429821 12154 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.211.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab.
Entry for principal kudu/127.3.211.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab.
I20260925 15:20:50.468815 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.193:0
--local_ip_for_outbound_sockets=127.3.211.193
--webserver_interface=127.3.211.193
--webserver_port=0
--tserver_master_addrs=127.3.211.254:38693
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab
--principal=kudu/127.3.211.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:42007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:20:50.559398 12166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:50.559638 12166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:50.559664 12166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:50.561169 12166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:50.561242 12166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:50.561275 12166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.193
I20260925 15:20:50.562953 12166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:42007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab
--principal=kudu/127.3.211.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.211.193
--webserver_port=0
--tserver_master_addrs=127.3.211.254:38693
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.193
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:50.563239 12166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:50.563508 12166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:50.569454 12171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:50.569411 12172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:50.569411 12174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:50.569751 12166 server_base.cc:1034] running on GCE node
I20260925 15:20:50.570042 12166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:50.570271 12166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:50.571434 12166 hybrid_clock.cc:648] HybridClock initialized: now 1790349650571402 us; error 53 us; skew 500 ppm
Sep 25 15:20:50 dist-test-slave-4bk3 krb5kdc[12059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349650, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:50.574173 12166 init.cc:377] Logged in from keytab as kudu/127.3.211.193@KRBTEST.COM (short username kudu)
I20260925 15:20:50.574646 12166 webserver.cc:460] Webserver started at http://127.3.211.193:43473/ using document root <none> and password file <none>
I20260925 15:20:50.574846 12166 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:50.574888 12166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:50.574976 12166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:50.576056 12166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-0/data/instance:
uuid: "216bc6751ec04e8f8e949ad64fbad716"
format_stamp: "Formatted at 2026-09-25 15:20:50 on dist-test-slave-4bk3"
server_key: "af75db37a801fbc172d55f74c21b94b8"
server_key_iv: "7aada6c22762c14b818460e4d3aa1c52"
server_key_version: "encryptionkey@0"
I20260925 15:20:50.576452 12166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-0/wal/instance:
uuid: "216bc6751ec04e8f8e949ad64fbad716"
format_stamp: "Formatted at 2026-09-25 15:20:50 on dist-test-slave-4bk3"
server_key: "af75db37a801fbc172d55f74c21b94b8"
server_key_iv: "7aada6c22762c14b818460e4d3aa1c52"
server_key_version: "encryptionkey@0"
I20260925 15:20:50.577834 12166 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:50.578711 12181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:50.578912 12166 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:50.579002 12166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-0/wal
uuid: "216bc6751ec04e8f8e949ad64fbad716"
format_stamp: "Formatted at 2026-09-25 15:20:50 on dist-test-slave-4bk3"
server_key: "af75db37a801fbc172d55f74c21b94b8"
server_key_iv: "7aada6c22762c14b818460e4d3aa1c52"
server_key_version: "encryptionkey@0"
I20260925 15:20:50.579054 12166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:50.596707 12166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:50.598920 12166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:50.599040 12166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:50.599249 12166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:20:50.599526 12166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:20:50.599558 12166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:50.599582 12166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:20:50.599596 12166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:50.606628 12166 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.193:37355
I20260925 15:20:50.606877 12294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.193:37355 every 8 connection(s)
I20260925 15:20:50.607131 12166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
Sep 25 15:20:50 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349650, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.193@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:20:50.615693 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 12166
I20260925 15:20:50.615815 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-0/wal/instance
I20260925 15:20:50.616021 3919 external_mini_cluster.cc:1428] Setting key 855ff11d822bd1eb58ff755ee831be92
I20260925 15:20:50.618925 12143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:50.608211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:56493 (local address 127.3.211.254:38693)
0925 15:20:50.608300 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:50.608302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:50.608673 (+ 371us) server_negotiation.cc:407] Connection header received
0925 15:20:50.609130 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:50.609132 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:50.609144 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:50.609216 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:50.610256 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:50.610837 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:50.611965 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:50.612207 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:50.612287 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:50.614982 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:50.614984 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:50.614984 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:50.614993 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:50.617181 (+ 2188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:50.617665 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:50.617667 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:50.617668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:50.617735 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:50.618059 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:50.618061 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:50.618062 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:50.618241 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:50.618309 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:50.618743 (+ 434us) server_negotiation.cc:299] Negotiation successful
0925 15:20:50.618851 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 15:20:50.619457 12297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:50.608189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38693 (local address 127.3.211.193:56493)
0925 15:20:50.608474 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:50.608521 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:50.609038 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:50.609276 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:50.609279 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:50.609875 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:50.610110 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:50.610115 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:50.610961 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:50.610962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:50.611843 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:50.611846 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:50.612447 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:50.612448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:50.612577 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:50.613201 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:50.613213 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:50.614844 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:50.617275 (+ 2431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:50.617287 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:50.617290 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:50.617566 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:50.617818 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:50.617819 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:50.617819 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:50.617942 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:50.618321 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:50.618328 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:50.618532 (+ 204us) client_negotiation.cc:769] Sending connection context
0925 15:20:50.618680 (+ 148us) client_negotiation.cc:240] Negotiation successful
0925 15:20:50.618898 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":164,"threads_started":1}
I20260925 15:20:50.620537 12295 heartbeater.cc:347] Connected to a master server at 127.3.211.254:38693
I20260925 15:20:50.620662 12295 heartbeater.cc:464] Registering TS with master...
I20260925 15:20:50.620913 12295 heartbeater.cc:511] Master 127.3.211.254:38693 requested a full tablet report, sending...
I20260925 15:20:50.621663 12104 ts_manager.cc:194] Registered new tserver with Master: 216bc6751ec04e8f8e949ad64fbad716 (127.3.211.193:37355)
I20260925 15:20:50.622722 12104 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.193@KRBTEST.COM'} at 127.3.211.193:56493
WARNING: no policy specified for kudu/127.3.211.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab.
Entry for principal kudu/127.3.211.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab.
I20260925 15:20:50.646911 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.194:0
--local_ip_for_outbound_sockets=127.3.211.194
--webserver_interface=127.3.211.194
--webserver_port=0
--tserver_master_addrs=127.3.211.254:38693
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab
--principal=kudu/127.3.211.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:42007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:20:50.738027 12300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:50.738277 12300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:50.738307 12300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:50.739624 12300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:50.739667 12300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:50.739698 12300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.194
I20260925 15:20:50.741410 12300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:42007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab
--principal=kudu/127.3.211.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.211.194
--webserver_port=0
--tserver_master_addrs=127.3.211.254:38693
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.194
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:50.741642 12300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:50.741952 12300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:50.747860 12306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:50.747872 12308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:50.747860 12305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:50.748167 12300 server_base.cc:1034] running on GCE node
I20260925 15:20:50.748477 12300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:50.748778 12300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:50.749946 12300 hybrid_clock.cc:648] HybridClock initialized: now 1790349650749919 us; error 44 us; skew 500 ppm
Sep 25 15:20:50 dist-test-slave-4bk3 krb5kdc[12059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349650, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:50.752976 12300 init.cc:377] Logged in from keytab as kudu/127.3.211.194@KRBTEST.COM (short username kudu)
I20260925 15:20:50.753545 12300 webserver.cc:460] Webserver started at http://127.3.211.194:34071/ using document root <none> and password file <none>
I20260925 15:20:50.753832 12300 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:50.753912 12300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:50.754074 12300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:50.755270 12300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-1/data/instance:
uuid: "ca14288a954147bea884eb337f4fa574"
format_stamp: "Formatted at 2026-09-25 15:20:50 on dist-test-slave-4bk3"
server_key: "07556d148911636b9fe3ea8da30f8a26"
server_key_iv: "eecf8805c581da921bc25a7ba6e8dea9"
server_key_version: "encryptionkey@0"
I20260925 15:20:50.755679 12300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-1/wal/instance:
uuid: "ca14288a954147bea884eb337f4fa574"
format_stamp: "Formatted at 2026-09-25 15:20:50 on dist-test-slave-4bk3"
server_key: "07556d148911636b9fe3ea8da30f8a26"
server_key_iv: "eecf8805c581da921bc25a7ba6e8dea9"
server_key_version: "encryptionkey@0"
I20260925 15:20:50.757385 12300 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260925 15:20:50.758322 12315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:50.758553 12300 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:50.758672 12300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-1/wal
uuid: "ca14288a954147bea884eb337f4fa574"
format_stamp: "Formatted at 2026-09-25 15:20:50 on dist-test-slave-4bk3"
server_key: "07556d148911636b9fe3ea8da30f8a26"
server_key_iv: "eecf8805c581da921bc25a7ba6e8dea9"
server_key_version: "encryptionkey@0"
I20260925 15:20:50.758780 12300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:50.786487 12300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:50.788960 12300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:50.789142 12300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:50.789393 12300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:20:50.789729 12300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:20:50.789788 12300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:50.789829 12300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:20:50.789856 12300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:50.796504 12300 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.194:41995
I20260925 15:20:50.796576 12428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.194:41995 every 8 connection(s)
I20260925 15:20:50.796972 12300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
I20260925 15:20:50.802915 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 12300
I20260925 15:20:50.803030 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-1/wal/instance
I20260925 15:20:50.803216 3919 external_mini_cluster.cc:1428] Setting key 2d7f473ea33b4941b5c9c0a78925a00c
Sep 25 15:20:50 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349650, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.194@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:20:50.807363 12143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:50.797778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:52867 (local address 127.3.211.254:38693)
0925 15:20:50.797830 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:50.797831 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:50.798036 (+ 205us) server_negotiation.cc:407] Connection header received
0925 15:20:50.798544 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:50.798545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:50.798556 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:50.798623 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:50.799591 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:50.800094 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:50.801335 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:50.801537 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:50.801600 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:50.804110 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:50.804112 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:50.804112 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:50.804121 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:50.805668 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:50.806213 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:50.806214 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:50.806215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:50.806284 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:50.806589 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:50.806591 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:50.806591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:50.806727 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:50.806835 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:50.807156 (+ 321us) server_negotiation.cc:299] Negotiation successful
0925 15:20:50.807251 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 15:20:50.807608 12431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:50.797739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38693 (local address 127.3.211.194:52867)
0925 15:20:50.797902 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:50.797945 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:50.798428 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:50.798701 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:50.798704 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:50.799279 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:50.799482 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:50.799486 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:50.800234 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:50.800235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:50.801236 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:50.801239 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:50.801644 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:50.801645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:50.801759 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:50.802262 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:50.802270 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:50.804011 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:50.805824 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:50.805834 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:50.805837 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:50.806111 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:50.806360 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:50.806361 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:50.806362 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:50.806491 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:50.806828 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:50.806836 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:50.807003 (+ 167us) client_negotiation.cc:769] Sending connection context
0925 15:20:50.807130 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 15:20:50.807333 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":86,"threads_started":1}
I20260925 15:20:50.808279 12429 heartbeater.cc:347] Connected to a master server at 127.3.211.254:38693
I20260925 15:20:50.808414 12429 heartbeater.cc:464] Registering TS with master...
I20260925 15:20:50.808609 12429 heartbeater.cc:511] Master 127.3.211.254:38693 requested a full tablet report, sending...
I20260925 15:20:50.809065 12104 ts_manager.cc:194] Registered new tserver with Master: ca14288a954147bea884eb337f4fa574 (127.3.211.194:41995)
I20260925 15:20:50.809659 12104 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.194@KRBTEST.COM'} at 127.3.211.194:52867
WARNING: no policy specified for kudu/127.3.211.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab.
Entry for principal kudu/127.3.211.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab.
I20260925 15:20:50.832708 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.195:0
--local_ip_for_outbound_sockets=127.3.211.195
--webserver_interface=127.3.211.195
--webserver_port=0
--tserver_master_addrs=127.3.211.254:38693
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab
--principal=kudu/127.3.211.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:42007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:20:50.917596 12434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:50.917819 12434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:50.917845 12434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:50.919142 12434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:50.919198 12434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:50.919227 12434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.195
I20260925 15:20:50.920883 12434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:42007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab
--principal=kudu/127.3.211.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.211.195
--webserver_port=0
--tserver_master_addrs=127.3.211.254:38693
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.195
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:50.921164 12434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:50.921429 12434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:50.926458 12442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:50.926458 12440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:50.926494 12439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:50.926855 12434 server_base.cc:1034] running on GCE node
I20260925 15:20:50.927028 12434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:50.927227 12434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:50.928421 12434 hybrid_clock.cc:648] HybridClock initialized: now 1790349650928399 us; error 27 us; skew 500 ppm
Sep 25 15:20:50 dist-test-slave-4bk3 krb5kdc[12059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349650, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:50.931351 12434 init.cc:377] Logged in from keytab as kudu/127.3.211.195@KRBTEST.COM (short username kudu)
I20260925 15:20:50.931843 12434 webserver.cc:460] Webserver started at http://127.3.211.195:44395/ using document root <none> and password file <none>
I20260925 15:20:50.932097 12434 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:50.932180 12434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:50.932374 12434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:20:50.933492 12434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-2/data/instance:
uuid: "a7641661254f47b89851af9598212635"
format_stamp: "Formatted at 2026-09-25 15:20:50 on dist-test-slave-4bk3"
server_key: "a31ccec8d9f6349fb57093e0d7ad3b1a"
server_key_iv: "2cf0d73e9fd73462a0e7b93aa87b0415"
server_key_version: "encryptionkey@0"
I20260925 15:20:50.933815 12434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-2/wal/instance:
uuid: "a7641661254f47b89851af9598212635"
format_stamp: "Formatted at 2026-09-25 15:20:50 on dist-test-slave-4bk3"
server_key: "a31ccec8d9f6349fb57093e0d7ad3b1a"
server_key_iv: "2cf0d73e9fd73462a0e7b93aa87b0415"
server_key_version: "encryptionkey@0"
I20260925 15:20:50.935280 12434 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:20:50.936177 12449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:50.936573 12434 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:50.936661 12434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-2/wal
uuid: "a7641661254f47b89851af9598212635"
format_stamp: "Formatted at 2026-09-25 15:20:50 on dist-test-slave-4bk3"
server_key: "a31ccec8d9f6349fb57093e0d7ad3b1a"
server_key_iv: "2cf0d73e9fd73462a0e7b93aa87b0415"
server_key_version: "encryptionkey@0"
I20260925 15:20:50.936806 12434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:50.960805 12434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:50.963596 12434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:50.963794 12434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:50.964113 12434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:20:50.964550 12434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:20:50.964589 12434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:50.964615 12434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:20:50.964628 12434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:50.971071 12434 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.195:33695
I20260925 15:20:50.971122 12562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.195:33695 every 8 connection(s)
I20260925 15:20:50.971583 12434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
I20260925 15:20:50.978295 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 12434
I20260925 15:20:50.978411 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-2/wal/instance
I20260925 15:20:50.978607 3919 external_mini_cluster.cc:1428] Setting key 8936e4e2f3dc1eb59f5ab9cafd871130
Sep 25 15:20:50 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349650, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.195@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:20:50.982509 12143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:50.972392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:36989 (local address 127.3.211.254:38693)
0925 15:20:50.972447 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:50.972450 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:50.972879 (+ 429us) server_negotiation.cc:407] Connection header received
0925 15:20:50.973480 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:50.973482 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:50.973496 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:50.973551 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:50.974597 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:50.975213 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:50.976447 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:50.976680 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:50.976786 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:50.979504 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:50.979506 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:50.979507 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:50.979515 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:50.980976 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:50.981434 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:50.981435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:50.981436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:50.981518 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:50.981837 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:50.981838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:50.981838 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:50.981978 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:50.982044 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:50.982354 (+ 310us) server_negotiation.cc:299] Negotiation successful
0925 15:20:50.982429 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 15:20:50.982728 12565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:50.972405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38693 (local address 127.3.211.195:36989)
0925 15:20:50.972745 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:50.972786 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:50.973356 (+ 570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:50.973623 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:50.973627 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:50.974272 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:50.974484 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:50.974489 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:50.975332 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:50.975333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:50.976224 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:50.976226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:50.976801 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:50.976802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:50.976953 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:50.977648 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:50.977658 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:50.979397 (+ 1739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:20:50.981085 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:50.981093 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:50.981096 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:50.981339 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:50.981612 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:20:50.981613 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:20:50.981614 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:20:50.981743 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:20:50.982060 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:20:50.982066 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:20:50.982198 (+ 132us) client_negotiation.cc:769] Sending connection context
0925 15:20:50.982300 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 15:20:50.982444 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":259,"threads_started":1}
I20260925 15:20:50.983246 12563 heartbeater.cc:347] Connected to a master server at 127.3.211.254:38693
I20260925 15:20:50.983337 12563 heartbeater.cc:464] Registering TS with master...
I20260925 15:20:50.983482 12563 heartbeater.cc:511] Master 127.3.211.254:38693 requested a full tablet report, sending...
I20260925 15:20:50.983821 12104 ts_manager.cc:194] Registered new tserver with Master: a7641661254f47b89851af9598212635 (127.3.211.195:33695)
I20260925 15:20:50.984174 12104 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.195@KRBTEST.COM'} at 127.3.211.195:36989
I20260925 15:20:50.990687 3919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.211.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.211.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.211.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/oryx_127.3.211.193.keytab.
Entry for principal oryx/127.3.211.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/oryx_127.3.211.193.keytab.
WARNING: no policy specified for oryx/127.3.211.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.211.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.211.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/oryx_127.3.211.194.keytab.
Entry for principal oryx/127.3.211.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/oryx_127.3.211.194.keytab.
WARNING: no policy specified for oryx/127.3.211.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.211.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.211.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/oryx_127.3.211.195.keytab.
Entry for principal oryx/127.3.211.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/oryx_127.3.211.195.keytab.
I20260925 15:20:51.076234 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 12166
I20260925 15:20:51.081158 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 12300
I20260925 15:20:51.085646 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 12434
I20260925 15:20:51.090489 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 12072
I20260925 15:20:51.096016 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:38693
--webserver_interface=127.3.211.254
--webserver_port=44619
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:42007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.211.254:38693
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:20:51.182456 12572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:51.182739 12572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:51.182790 12572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:51.184118 12572 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:20:51.184204 12572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:51.184229 12572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:51.184250 12572 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:20:51.184284 12572 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:20:51.185952 12572 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:42007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.211.254:38693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:38693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.211.254
--webserver_port=44619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:51.186266 12572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:51.186528 12572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:51.191876 12577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:51.191910 12580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:51.192129 12572 server_base.cc:1034] running on GCE node
W20260925 15:20:51.192235 12578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:51.192560 12572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:51.192955 12572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:51.194229 12572 hybrid_clock.cc:648] HybridClock initialized: now 1790349651194171 us; error 64 us; skew 500 ppm
Sep 25 15:20:51 dist-test-slave-4bk3 krb5kdc[12059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349651, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:51.197875 12572 init.cc:377] Logged in from keytab as kudu/127.3.211.254@KRBTEST.COM (short username kudu)
I20260925 15:20:51.198360 12572 webserver.cc:460] Webserver started at http://127.3.211.254:44619/ using document root <none> and password file <none>
I20260925 15:20:51.198632 12572 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:51.198701 12572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:51.200295 12572 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 15:20:51.201224 12587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:51.201658 12572 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:20:51.201756 12572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/master-0/wal
uuid: "5b95d329f4c14b61be453c8c690c4110"
format_stamp: "Formatted at 2026-09-25 15:20:50 on dist-test-slave-4bk3"
server_key: "b9c6a85c2572166eb8fd502b281fd322"
server_key_iv: "43c3e7b9b4b8e67290bddf210e213684"
server_key_version: "encryptionkey@0"
I20260925 15:20:51.202179 12572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:51.213490 12572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:51.215885 12572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:51.216037 12572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:51.220103 12572 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.254:38693
I20260925 15:20:51.220156 12639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.254:38693 every 8 connection(s)
I20260925 15:20:51.220597 12572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
I20260925 15:20:51.221549 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 12572
I20260925 15:20:51.221871 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.193:37355
--local_ip_for_outbound_sockets=127.3.211.193
--tserver_master_addrs=127.3.211.254:38693
--webserver_port=43473
--webserver_interface=127.3.211.193
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab
--principal=kudu/127.3.211.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:42007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.211.193
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/oryx_127.3.211.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:20:51.222616 12640 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:20:51.223776 12640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b95d329f4c14b61be453c8c690c4110: Bootstrap starting.
I20260925 15:20:51.232540 12640 log.cc:824] T 00000000000000000000000000000000 P 5b95d329f4c14b61be453c8c690c4110: Log is configured to *not* fsync() on all Append() calls
I20260925 15:20:51.234423 12640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b95d329f4c14b61be453c8c690c4110: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:20:51.234784 12640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b95d329f4c14b61be453c8c690c4110: Bootstrap complete.
I20260925 15:20:51.237246 12640 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b95d329f4c14b61be453c8c690c4110 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b95d329f4c14b61be453c8c690c4110" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38693 } }
I20260925 15:20:51.237488 12640 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b95d329f4c14b61be453c8c690c4110 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b95d329f4c14b61be453c8c690c4110, State: Initialized, Role: FOLLOWER
I20260925 15:20:51.237666 12640 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b95d329f4c14b61be453c8c690c4110 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b95d329f4c14b61be453c8c690c4110" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38693 } }
I20260925 15:20:51.237739 12640 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b95d329f4c14b61be453c8c690c4110 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:20:51.237793 12640 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b95d329f4c14b61be453c8c690c4110 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:20:51.237843 12640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b95d329f4c14b61be453c8c690c4110 [term 1 FOLLOWER]: Advancing to term 2
I20260925 15:20:51.238895 12640 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b95d329f4c14b61be453c8c690c4110 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b95d329f4c14b61be453c8c690c4110" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38693 } }
I20260925 15:20:51.238984 12640 leader_election.cc:302] T 00000000000000000000000000000000 P 5b95d329f4c14b61be453c8c690c4110 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b95d329f4c14b61be453c8c690c4110; no voters:
I20260925 15:20:51.239207 12640 leader_election.cc:288] T 00000000000000000000000000000000 P 5b95d329f4c14b61be453c8c690c4110 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 15:20:51.239246 12644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b95d329f4c14b61be453c8c690c4110 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 15:20:51.239463 12644 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b95d329f4c14b61be453c8c690c4110 [term 2 LEADER]: Becoming Leader. State: Replica: 5b95d329f4c14b61be453c8c690c4110, State: Running, Role: LEADER
I20260925 15:20:51.239497 12640 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b95d329f4c14b61be453c8c690c4110 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:20:51.239615 12644 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b95d329f4c14b61be453c8c690c4110 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b95d329f4c14b61be453c8c690c4110" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38693 } }
I20260925 15:20:51.239950 12645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b95d329f4c14b61be453c8c690c4110 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5b95d329f4c14b61be453c8c690c4110" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b95d329f4c14b61be453c8c690c4110" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38693 } } }
I20260925 15:20:51.239962 12646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b95d329f4c14b61be453c8c690c4110 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b95d329f4c14b61be453c8c690c4110. Latest consensus state: current_term: 2 leader_uuid: "5b95d329f4c14b61be453c8c690c4110" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b95d329f4c14b61be453c8c690c4110" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38693 } } }
I20260925 15:20:51.240051 12645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b95d329f4c14b61be453c8c690c4110 [sys.catalog]: This master's current role is: LEADER
I20260925 15:20:51.240069 12646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b95d329f4c14b61be453c8c690c4110 [sys.catalog]: This master's current role is: LEADER
I20260925 15:20:51.240245 12650 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:20:51.240859 12650 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:20:51.241067 12650 catalog_manager.cc:1259] Loaded cluster ID: 0b3f68beb6de4bbd973af673a50f42e4
I20260925 15:20:51.241235 12650 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:20:51.242417 12650 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:20:51.242584 12650 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5b95d329f4c14b61be453c8c690c4110: Loaded TSK: 0
I20260925 15:20:51.242791 12650 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 15:20:51.310266 12642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:51.310519 12642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:51.310549 12642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:51.311914 12642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:51.311973 12642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:51.312003 12642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.193
I20260925 15:20:51.313752 12642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:42007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/oryx_127.3.211.193.keytab
--principal=oryx/127.3.211.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.193:37355
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.211.193
--webserver_port=43473
--tserver_master_addrs=127.3.211.254:38693
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.193
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:51.314049 12642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:51.314321 12642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:51.319591 12667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:51.319591 12666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:51.319705 12669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:51.319973 12642 server_base.cc:1034] running on GCE node
I20260925 15:20:51.320201 12642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:51.320518 12642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:51.321707 12642 hybrid_clock.cc:648] HybridClock initialized: now 1790349651321685 us; error 36 us; skew 500 ppm
Sep 25 15:20:51 dist-test-slave-4bk3 krb5kdc[12059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349651, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.211.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:51.324919 12642 init.cc:377] Logged in from keytab as oryx/127.3.211.193@KRBTEST.COM (short username oryx)
I20260925 15:20:51.325439 12642 webserver.cc:460] Webserver started at http://127.3.211.193:43473/ using document root <none> and password file <none>
I20260925 15:20:51.325700 12642 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:51.325788 12642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:51.327642 12642 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:20:51.328450 12676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:51.328637 12642 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 15:20:51.328737 12642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-0/wal
uuid: "216bc6751ec04e8f8e949ad64fbad716"
format_stamp: "Formatted at 2026-09-25 15:20:50 on dist-test-slave-4bk3"
server_key: "af75db37a801fbc172d55f74c21b94b8"
server_key_iv: "7aada6c22762c14b818460e4d3aa1c52"
server_key_version: "encryptionkey@0"
I20260925 15:20:51.329182 12642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:51.363718 12642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:51.366686 12642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:51.366875 12642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:51.367170 12642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:20:51.367574 12642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:20:51.367628 12642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:51.367663 12642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:20:51.367689 12642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:51.375536 12642 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.193:37355
I20260925 15:20:51.375609 12789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.193:37355 every 8 connection(s)
I20260925 15:20:51.376024 12642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:20:51.378960 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 12642
I20260925 15:20:51.379271 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.194:41995
--local_ip_for_outbound_sockets=127.3.211.194
--tserver_master_addrs=127.3.211.254:38693
--webserver_port=34071
--webserver_interface=127.3.211.194
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab
--principal=kudu/127.3.211.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:42007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.211.194
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/oryx_127.3.211.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:20:51 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.193@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:51 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.193@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:20:51.387574 12792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:51.377041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38693 (local address 127.3.211.193:41621)
0925 15:20:51.377226 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:51.377272 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:51.377759 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:51.378318 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:51.378322 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:51.379172 (+ 850us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:51.379603 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:51.379611 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:51.381514 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:51.381515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:51.382380 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:51.382383 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:51.383233 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:51.383234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:51.383402 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:51.384069 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:51.384081 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:51.387289 (+ 3208us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.211.254:38693: Server oryx/127.3.211.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":103,"threads_started":1}
W20260925 15:20:51.387827 12790 heartbeater.cc:644] Failed to heartbeat to 127.3.211.254:38693 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.211.254:38693: Client connection negotiation failed: client connection to 127.3.211.254:38693: Server oryx/127.3.211.254@KRBTEST.COM not found in Kerberos database
W20260925 15:20:51.388581 12793 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:51.377135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:41621 (local address 127.3.211.254:38693)
0925 15:20:51.377959 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:51.377966 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:51.377991 (+ 25us) server_negotiation.cc:407] Connection header received
0925 15:20:51.378092 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:51.378096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:51.378135 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:51.378282 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:51.379822 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:51.381371 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:51.382493 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:51.383109 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:51.383247 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:51.388313 (+ 5066us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.211.193:41621: BlockingRecv error: recv got EOF from 127.3.211.193:41621 (error 108)
Metrics: {"server-negotiator.queue_time_us":774,"thread_start_us":88,"threads_started":1}
Sep 25 15:20:51 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.193@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:51 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.193@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:20:51.394083 12792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:51.388115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38693 (local address 127.3.211.193:46439)
0925 15:20:51.388174 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:51.388185 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:51.388231 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:51.388750 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:51.388751 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:51.388939 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:51.389010 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:51.389013 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:51.389904 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:51.389905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:51.390892 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:51.390895 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:51.391374 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:51.391375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:51.391454 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:51.391979 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:51.391985 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:51.393974 (+ 1989us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.211.254:38693: Server oryx/127.3.211.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 15:20:51.394176 12795 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:51.388151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:46439 (local address 127.3.211.254:38693)
0925 15:20:51.388541 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:51.388543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:51.388554 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:20:51.388583 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:51.388584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:51.388597 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:51.388689 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:51.389113 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:51.389798 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:51.390998 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:51.391282 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:51.391357 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:51.394100 (+ 2743us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.211.193:46439: BlockingRecv error: recv got EOF from 127.3.211.193:46439 (error 108)
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":110,"threads_started":1}
Sep 25 15:20:51 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.193@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:51 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.193@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:20:51.399288 12792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:51.394412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38693 (local address 127.3.211.193:37283)
0925 15:20:51.394451 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:51.394464 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:51.394514 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:51.394649 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:51.394650 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:51.394770 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:51.394802 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:51.394804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:51.395750 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:51.395751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:51.396395 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:51.396398 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:51.396924 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:51.396925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:51.396973 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:51.397489 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:51.397495 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:51.399231 (+ 1736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.211.254:38693: Server oryx/127.3.211.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":19}
W20260925 15:20:51.399451 12790 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 15:20:51.399473 12795 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:51.394476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:37283 (local address 127.3.211.254:38693)
0925 15:20:51.394535 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:51.394537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:51.394548 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:20:51.394573 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:51.394574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:51.394584 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:51.394653 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:51.394866 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:51.395633 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:51.396500 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:51.396824 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:51.396894 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:51.399347 (+ 2453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.211.193:37283: BlockingRecv error: recv got EOF from 127.3.211.193:37283 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260925 15:20:51.469915 12794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:51.470142 12794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:51.470168 12794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:51.471338 12794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:51.471383 12794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:51.471412 12794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.194
I20260925 15:20:51.472950 12794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:42007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/oryx_127.3.211.194.keytab
--principal=oryx/127.3.211.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.194:41995
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.211.194
--webserver_port=34071
--tserver_master_addrs=127.3.211.254:38693
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.194
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:51.473204 12794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:51.473397 12794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:51.478690 12801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:51.478678 12800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:51.478655 12803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:51.479032 12794 server_base.cc:1034] running on GCE node
I20260925 15:20:51.479276 12794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:51.479616 12794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:51.480814 12794 hybrid_clock.cc:648] HybridClock initialized: now 1790349651480809 us; error 43 us; skew 500 ppm
Sep 25 15:20:51 dist-test-slave-4bk3 krb5kdc[12059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349651, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.211.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:51.483666 12794 init.cc:377] Logged in from keytab as oryx/127.3.211.194@KRBTEST.COM (short username oryx)
I20260925 15:20:51.484192 12794 webserver.cc:460] Webserver started at http://127.3.211.194:34071/ using document root <none> and password file <none>
I20260925 15:20:51.484508 12794 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:51.484602 12794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:51.486215 12794 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:51.487010 12810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:51.487246 12794 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:51.487429 12794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-1/wal
uuid: "ca14288a954147bea884eb337f4fa574"
format_stamp: "Formatted at 2026-09-25 15:20:50 on dist-test-slave-4bk3"
server_key: "07556d148911636b9fe3ea8da30f8a26"
server_key_iv: "eecf8805c581da921bc25a7ba6e8dea9"
server_key_version: "encryptionkey@0"
I20260925 15:20:51.487849 12794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:51.518110 12794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:51.521064 12794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:51.521279 12794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:51.521548 12794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:20:51.521881 12794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:20:51.521950 12794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:51.521994 12794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:20:51.522032 12794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:51.528404 12794 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.194:41995
I20260925 15:20:51.528448 12923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.194:41995 every 8 connection(s)
I20260925 15:20:51.528820 12794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
Sep 25 15:20:51 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.194@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:51 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.194@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:20:51.536614 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 12794
I20260925 15:20:51.536895 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.195:33695
--local_ip_for_outbound_sockets=127.3.211.195
--tserver_master_addrs=127.3.211.254:38693
--webserver_port=44395
--webserver_interface=127.3.211.195
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab
--principal=kudu/127.3.211.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:42007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.211.195
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/oryx_127.3.211.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:20:51.537308 12926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:51.529598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38693 (local address 127.3.211.194:40123)
0925 15:20:51.529761 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:51.529804 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:51.530279 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:51.530561 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:51.530565 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:51.531141 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:51.531389 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:51.531394 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:51.532175 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:51.532176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:51.533212 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:51.533214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:51.533657 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:51.533659 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:51.533783 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:51.534427 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:51.534437 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:51.537134 (+ 2697us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.211.254:38693: Server oryx/127.3.211.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":88,"threads_started":1}
W20260925 15:20:51.537348 12795 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:51.529607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:40123 (local address 127.3.211.254:38693)
0925 15:20:51.529679 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:51.529681 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:51.529898 (+ 217us) server_negotiation.cc:407] Connection header received
0925 15:20:51.530410 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:51.530412 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:51.530427 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:51.530503 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:51.531512 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:51.532060 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:51.533319 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:51.533525 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:51.533594 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:51.537259 (+ 3665us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.211.194:40123: BlockingRecv error: recv got EOF from 127.3.211.194:40123 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260925 15:20:51.537575 12924 heartbeater.cc:644] Failed to heartbeat to 127.3.211.254:38693 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.211.254:38693: Client connection negotiation failed: client connection to 127.3.211.254:38693: Server oryx/127.3.211.254@KRBTEST.COM not found in Kerberos database
Sep 25 15:20:51 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.194@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:51 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.194@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:20:51.542718 12926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:51.537847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38693 (local address 127.3.211.194:37383)
0925 15:20:51.537902 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:51.537911 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:51.537967 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:51.538122 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:51.538123 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:51.538261 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:51.538316 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:51.538317 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:51.538980 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:51.538982 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:51.539488 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:51.539489 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:51.539958 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:51.539959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:51.540031 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:51.540564 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:51.540570 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:51.542624 (+ 2054us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.211.254:38693: Server oryx/127.3.211.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 15:20:51.542794 12795 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:51.537906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:37383 (local address 127.3.211.254:38693)
0925 15:20:51.537964 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:51.537967 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:51.537976 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:20:51.538029 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:51.538031 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:51.538044 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:51.538120 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:51.538397 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:51.538886 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:51.539628 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:51.539883 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:51.539961 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:51.542713 (+ 2752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.211.194:37383: BlockingRecv error: recv got EOF from 127.3.211.194:37383 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 25 15:20:51 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.194@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:51 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.194@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:20:51.547931 12926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:51.543131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38693 (local address 127.3.211.194:58417)
0925 15:20:51.543284 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:51.543292 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:51.543364 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:51.543511 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:51.543512 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:51.543641 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:51.543676 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:51.543678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:51.544359 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:51.544360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:51.544861 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:51.544863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:51.545219 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:51.545220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:51.545269 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:51.545782 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:51.545786 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:51.547851 (+ 2065us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.211.254:38693: Server oryx/127.3.211.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":104,"threads_started":1}
W20260925 15:20:51.548038 12795 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:51.543171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:58417 (local address 127.3.211.254:38693)
0925 15:20:51.543215 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:51.543217 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:51.543348 (+ 131us) server_negotiation.cc:407] Connection header received
0925 15:20:51.543442 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:51.543443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:51.543454 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:51.543499 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:51.543750 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:51.544249 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:51.544956 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:51.545144 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:51.545221 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:51.547953 (+ 2732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.211.194:58417: BlockingRecv error: recv got EOF from 127.3.211.194:58417 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260925 15:20:51.548094 12924 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 15:20:51.620010 12927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:20:51.620242 12927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:20:51.620270 12927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:20:51.621533 12927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:20:51.621591 12927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:20:51.621621 12927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.195
I20260925 15:20:51.623178 12927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:42007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/krb5kdc/oryx_127.3.211.195.keytab
--principal=oryx/127.3.211.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.195:33695
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.211.195
--webserver_port=44395
--tserver_master_addrs=127.3.211.254:38693
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.195
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:20:51.623462 12927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:20:51.623723 12927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:20:51.629187 12936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:51.629187 12934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:20:51.629190 12933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:20:51.629467 12927 server_base.cc:1034] running on GCE node
I20260925 15:20:51.629700 12927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:20:51.629925 12927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:20:51.631090 12927 hybrid_clock.cc:648] HybridClock initialized: now 1790349651631081 us; error 35 us; skew 500 ppm
Sep 25 15:20:51 dist-test-slave-4bk3 krb5kdc[12059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349651, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.211.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:20:51.633905 12927 init.cc:377] Logged in from keytab as oryx/127.3.211.195@KRBTEST.COM (short username oryx)
I20260925 15:20:51.634449 12927 webserver.cc:460] Webserver started at http://127.3.211.195:44395/ using document root <none> and password file <none>
I20260925 15:20:51.634691 12927 fs_manager.cc:359] Metadata directory not provided
I20260925 15:20:51.634763 12927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:20:51.636300 12927 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:20:51.637109 12943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:51.637331 12927 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:20:51.637428 12927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-2/wal
uuid: "a7641661254f47b89851af9598212635"
format_stamp: "Formatted at 2026-09-25 15:20:50 on dist-test-slave-4bk3"
server_key: "a31ccec8d9f6349fb57093e0d7ad3b1a"
server_key_iv: "2cf0d73e9fd73462a0e7b93aa87b0415"
server_key_version: "encryptionkey@0"
I20260925 15:20:51.637789 12927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:20:51.656126 12927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:20:51.658670 12927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:20:51.658802 12927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:20:51.658993 12927 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:20:51.659430 12927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:20:51.659476 12927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:51.659502 12927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:20:51.659514 12927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:20:51.665408 12927 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.195:33695
I20260925 15:20:51.665478 13056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.195:33695 every 8 connection(s)
I20260925 15:20:51.665846 12927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
Sep 25 15:20:51 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.195@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:51 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.195@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:20:51.672912 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 12927
W20260925 15:20:51.673444 12795 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:51.666673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:41235 (local address 127.3.211.254:38693)
0925 15:20:51.666718 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:51.666720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:51.666952 (+ 232us) server_negotiation.cc:407] Connection header received
0925 15:20:51.667421 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:51.667422 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:51.667434 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:51.667489 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:51.668464 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:51.668968 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:51.670025 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:51.670218 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:51.670283 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:51.673383 (+ 3100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.211.195:41235: BlockingRecv error: recv got EOF from 127.3.211.195:41235 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 15:20:51.673520 13059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:51.666660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38693 (local address 127.3.211.195:41235)
0925 15:20:51.666819 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:51.666855 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:51.667320 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:51.667539 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:51.667542 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:51.668098 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:51.668302 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:51.668306 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:51.669091 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:51.669092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:51.669907 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:51.669910 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:51.670306 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:51.670307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:51.670424 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:51.670952 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:51.670961 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:51.673309 (+ 2348us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.211.254:38693: Server oryx/127.3.211.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":99,"threads_started":1}
W20260925 15:20:51.673748 13057 heartbeater.cc:644] Failed to heartbeat to 127.3.211.254:38693 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.211.254:38693: Client connection negotiation failed: client connection to 127.3.211.254:38693: Server oryx/127.3.211.254@KRBTEST.COM not found in Kerberos database
Sep 25 15:20:51 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.195@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:51 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.195@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:20:51.679971 13059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:51.674011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38693 (local address 127.3.211.195:47361)
0925 15:20:51.674059 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:51.674068 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:51.674118 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:51.674254 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:51.674255 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:51.674391 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:51.674435 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:51.674436 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:51.675415 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:51.675417 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:51.676244 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:51.676247 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:51.676829 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:51.676829 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:51.676897 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:51.677419 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:51.677442 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:51.679893 (+ 2451us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.211.254:38693: Server oryx/127.3.211.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 15:20:51.680136 12793 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:51.674060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:47361 (local address 127.3.211.254:38693)
0925 15:20:51.674116 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:51.674118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:51.674127 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:20:51.674183 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:51.674184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:51.674197 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:51.674243 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:51.674509 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:51.675307 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:51.676420 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:51.676753 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:51.676817 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:51.680011 (+ 3194us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.211.195:47361: BlockingRecv error: recv got EOF from 127.3.211.195:47361 (error 108)
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":35}
I20260925 15:20:51.682502 12795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:51.673502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35430 (local address 127.3.211.254:38693)
0925 15:20:51.673589 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:51.673591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:51.673669 (+ 78us) server_negotiation.cc:407] Connection header received
0925 15:20:51.673762 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:51.673763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:51.673774 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:51.673826 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:51.674538 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:51.675307 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:51.676437 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:51.676733 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:51.676817 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:51.678422 (+ 1605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:20:51.678429 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:20:51.678429 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:20:51.678446 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:20:51.680783 (+ 2337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:51.681228 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:51.681231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:51.681231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:51.681336 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:20:51.681586 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:20:51.681587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:20:51.681587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:20:51.681953 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:20:51.682147 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 15:20:51.682256 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 15:20:51.682379 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":62}
Sep 25 15:20:51 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.195@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:51 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.195@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:20:51.685335 13059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:51.680289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38693 (local address 127.3.211.195:39415)
0925 15:20:51.680353 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:51.680363 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:51.680400 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:51.680608 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:51.680609 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:51.680756 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:51.680802 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:51.680804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:51.681675 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:51.681676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:51.682435 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:51.682438 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:51.682934 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:51.682935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:51.682992 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:51.683499 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:51.683504 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:51.685282 (+ 1778us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.211.254:38693: Server oryx/127.3.211.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260925 15:20:51.685477 12793 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:51.680425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:39415 (local address 127.3.211.254:38693)
0925 15:20:51.680481 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:51.680483 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:51.680491 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:20:51.680520 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:51.680521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:51.680531 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:51.680581 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:51.680893 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:51.681581 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:51.682538 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:51.682856 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:51.682924 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:51.685403 (+ 2479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.211.195:39415: BlockingRecv error: recv got EOF from 127.3.211.195:39415 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260925 15:20:51.685480 13057 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 15:20:52 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.193@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:52 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.193@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:20:52.404681 13061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:52.399937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38693 (local address 127.3.211.193:33841)
0925 15:20:52.400105 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:52.400118 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:52.400198 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:52.400395 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:52.400395 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:52.400559 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:52.400607 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:52.400608 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:52.401279 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:52.401280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:52.401957 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:52.401959 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:52.402366 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:52.402367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:52.402407 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:52.402823 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:52.402827 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:52.404611 (+ 1784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.211.254:38693: Server oryx/127.3.211.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":56,"threads_started":1}
W20260925 15:20:52.404834 13062 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:52.400068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:33841 (local address 127.3.211.254:38693)
0925 15:20:52.400198 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:52.400200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:52.400211 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:20:52.400256 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:52.400257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:52.400267 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:52.400347 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:52.400682 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:52.401196 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:52.402024 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:52.402299 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:52.402365 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:52.404701 (+ 2336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.211.193:33841: BlockingRecv error: recv got EOF from 127.3.211.193:33841 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":64,"threads_started":1}
Sep 25 15:20:52 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.194@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:52 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.194@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:20:52.553524 13063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:52.548605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38693 (local address 127.3.211.194:36529)
0925 15:20:52.548767 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:52.548779 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:52.548855 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:52.549020 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:52.549021 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:52.549182 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:52.549230 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:52.549232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:52.549920 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:52.549921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:52.550554 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:52.550556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:52.550964 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:52.550966 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:52.551016 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:52.551600 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:52.551605 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:52.553451 (+ 1846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.211.254:38693: Server oryx/127.3.211.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":69,"threads_started":1}
W20260925 15:20:52.553709 13062 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:52.548713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:36529 (local address 127.3.211.254:38693)
0925 15:20:52.548791 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:52.548793 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:52.548900 (+ 107us) server_negotiation.cc:407] Connection header received
0925 15:20:52.548925 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:52.548926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:52.548936 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:52.548978 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:52.549311 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:52.549820 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:52.550665 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:52.550869 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:52.550937 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:52.553640 (+ 2703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.211.194:36529: BlockingRecv error: recv got EOF from 127.3.211.194:36529 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 25 15:20:52 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.195@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:52 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.195@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:20:52.690717 13064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:52.685968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38693 (local address 127.3.211.195:45155)
0925 15:20:52.686110 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:52.686120 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:52.686179 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:52.686346 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:52.686346 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:52.686516 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:52.686565 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:52.686567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:52.687214 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:52.687214 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:52.687770 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:52.687773 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:52.688157 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:52.688158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:52.688208 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:52.688701 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:52.688706 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:52.690646 (+ 1940us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.211.254:38693: Server oryx/127.3.211.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":68,"threads_started":1}
W20260925 15:20:52.690848 13062 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:52.686067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:45155 (local address 127.3.211.254:38693)
0925 15:20:52.686132 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:52.686134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:52.686164 (+ 30us) server_negotiation.cc:407] Connection header received
0925 15:20:52.686243 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:52.686244 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:52.686255 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:52.686328 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:52.686629 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:52.687153 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:52.687866 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:52.688058 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:52.688127 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:52.690768 (+ 2641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.211.195:45155: BlockingRecv error: recv got EOF from 127.3.211.195:45155 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 25 15:20:53 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.193@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:53 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.193@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:20:53.410266 13065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:53.405221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38693 (local address 127.3.211.193:41773)
0925 15:20:53.405417 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:53.405466 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:53.405548 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:53.405733 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:53.405743 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:53.405904 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:53.405956 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:53.405957 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:53.406724 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:53.406725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:53.407330 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:53.407331 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:53.407765 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:53.407766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:53.407817 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:53.408304 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:53.408309 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:53.410204 (+ 1895us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.211.254:38693: Server oryx/127.3.211.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":91,"threads_started":1}
W20260925 15:20:53.410511 13066 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:53.405338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:41773 (local address 127.3.211.254:38693)
0925 15:20:53.405480 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:53.405482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:53.405537 (+ 55us) server_negotiation.cc:407] Connection header received
0925 15:20:53.405596 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:53.405597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:53.405607 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:53.405668 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:53.406031 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:53.406586 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:53.407446 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:53.407654 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:53.407738 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:53.410301 (+ 2563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.211.193:41773: BlockingRecv error: recv got EOF from 127.3.211.193:41773 (error 108)
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":77,"threads_started":1}
Sep 25 15:20:53 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.194@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:53 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.194@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:20:53.559201 13067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:53.554083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38693 (local address 127.3.211.194:44379)
0925 15:20:53.554211 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:53.554223 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:53.554286 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:53.554507 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:53.554508 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:53.554675 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:53.554726 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:53.554728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:53.555446 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:53.555447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:53.556026 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:53.556028 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:53.556535 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:53.556537 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:53.556612 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:53.557275 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:53.557281 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:53.559137 (+ 1856us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.211.254:38693: Server oryx/127.3.211.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":75,"threads_started":1}
W20260925 15:20:53.559382 13066 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:53.554245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:44379 (local address 127.3.211.254:38693)
0925 15:20:53.554325 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:53.554327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:53.554339 (+ 12us) server_negotiation.cc:407] Connection header received
0925 15:20:53.554369 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:53.554370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:53.554383 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:53.554458 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:53.554826 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:53.555343 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:53.556143 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:53.556415 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:53.556496 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:53.559290 (+ 2794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.211.194:44379: BlockingRecv error: recv got EOF from 127.3.211.194:44379 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 25 15:20:53 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.195@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:53 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.195@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:20:53.696055 13068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:53.691343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38693 (local address 127.3.211.195:42981)
0925 15:20:53.691480 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:53.691498 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:53.691545 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:53.691798 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:53.691800 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:53.691962 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:53.692014 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:53.692016 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:53.692671 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:53.692672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:53.693193 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:53.693195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:53.693538 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:53.693539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:53.693586 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:53.694050 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:53.694055 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:53.695993 (+ 1938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.211.254:38693: Server oryx/127.3.211.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":92,"threads_started":1}
W20260925 15:20:53.696244 13066 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:53.691545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:42981 (local address 127.3.211.254:38693)
0925 15:20:53.691613 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:53.691620 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:53.691628 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:20:53.691659 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:53.691659 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:53.691670 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:53.691729 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:53.692087 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:53.692602 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:53.693271 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:53.693454 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:53.693510 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:53.696172 (+ 2662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.211.195:42981: BlockingRecv error: recv got EOF from 127.3.211.195:42981 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 25 15:20:54 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.193@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:54 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.193@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:20:54.415717 13069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:54.410858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38693 (local address 127.3.211.193:52819)
0925 15:20:54.411008 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:54.411018 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:54.411067 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:54.411221 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:54.411222 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:54.411383 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:54.411432 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:54.411434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:54.412106 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:54.412107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:54.412774 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:54.412777 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:54.413168 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:54.413169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:54.413207 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:54.413714 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:54.413720 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:54.415654 (+ 1934us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.211.254:38693: Server oryx/127.3.211.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":91,"threads_started":1}
W20260925 15:20:54.415933 13070 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:54.410963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:52819 (local address 127.3.211.254:38693)
0925 15:20:54.411093 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:54.411095 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:54.411102 (+ 7us) server_negotiation.cc:407] Connection header received
0925 15:20:54.411124 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:54.411124 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:54.411134 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:54.411201 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:54.411503 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:54.412017 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:54.412874 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:54.413079 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:54.413161 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:54.415847 (+ 2686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.211.193:52819: BlockingRecv error: recv got EOF from 127.3.211.193:52819 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":76,"threads_started":1}
Sep 25 15:20:54 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.194@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:54 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.194@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:20:54.564921 13071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:54.559760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38693 (local address 127.3.211.194:56211)
0925 15:20:54.559902 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:54.559916 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:54.560075 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:54.560277 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:54.560278 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:54.560464 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:54.560512 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:54.560514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:54.561436 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:54.561437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:54.561963 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:54.561965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:54.562336 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:54.562337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:54.562379 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:54.562986 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:54.562990 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:54.564861 (+ 1871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.211.254:38693: Server oryx/127.3.211.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":91,"threads_started":1}
W20260925 15:20:54.565104 13070 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:54.559840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:56211 (local address 127.3.211.254:38693)
0925 15:20:54.559931 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:54.559933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:54.560072 (+ 139us) server_negotiation.cc:407] Connection header received
0925 15:20:54.560125 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:54.560125 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:54.560136 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:54.560192 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:54.560614 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:54.561117 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:54.562069 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:54.562264 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:54.562324 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:54.565025 (+ 2701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.211.194:56211: BlockingRecv error: recv got EOF from 127.3.211.194:56211 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 25 15:20:54 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.195@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:54 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.195@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:20:54.701678 13072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:54.696737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38693 (local address 127.3.211.195:43881)
0925 15:20:54.696895 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:54.696907 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:54.696981 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:54.697165 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:54.697166 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:54.697320 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:54.697377 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:54.697378 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:54.698160 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:54.698161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:54.698690 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:54.698692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:54.699065 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:54.699066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:54.699109 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:54.699633 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:54.699638 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:54.701573 (+ 1935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.211.254:38693: Server oryx/127.3.211.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":113,"threads_started":1}
W20260925 15:20:54.701722 13070 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:54.696808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:43881 (local address 127.3.211.254:38693)
0925 15:20:54.696890 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:54.696892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:54.697009 (+ 117us) server_negotiation.cc:407] Connection header received
0925 15:20:54.697042 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:54.697043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:54.697061 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:54.697147 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:54.697500 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:54.698066 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:54.698775 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:54.698986 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:54.699053 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:54.701654 (+ 2601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.211.195:43881: BlockingRecv error: recv got EOF from 127.3.211.195:43881 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 25 15:20:55 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.193@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:55 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.193@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:20:55.425971 13073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:55.416360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38693 (local address 127.3.211.193:50949)
0925 15:20:55.416495 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:55.416513 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:55.416565 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:55.416943 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:55.416945 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:55.417137 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:55.417199 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:55.417201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:55.417960 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:55.417962 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:55.418593 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:55.418596 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:55.419043 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:55.419045 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:55.419097 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:55.423615 (+ 4518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:55.423621 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:55.425909 (+ 2288us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.211.254:38693: Server oryx/127.3.211.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":82,"threads_started":1}
W20260925 15:20:55.426131 13074 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:55.416611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:50949 (local address 127.3.211.254:38693)
0925 15:20:55.416746 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:55.416748 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:55.416764 (+ 16us) server_negotiation.cc:407] Connection header received
0925 15:20:55.416788 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:55.416789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:55.416799 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:55.416876 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:55.417329 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:55.417847 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:55.418709 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:55.418927 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:55.418994 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:55.426029 (+ 7035us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.211.193:50949: BlockingRecv error: recv got EOF from 127.3.211.193:50949 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":88,"threads_started":1}
Sep 25 15:20:55 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.194@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:55 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.194@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:20:55.570497 13075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:55.565518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38693 (local address 127.3.211.194:52067)
0925 15:20:55.565635 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:55.565649 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:55.565706 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:55.565857 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:55.565857 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:55.566018 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:55.566067 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:55.566069 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:55.566834 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:55.566835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:55.567365 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:55.567367 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:55.567748 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:55.567756 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:55.567804 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:55.568396 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:55.568400 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:55.570435 (+ 2035us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.211.254:38693: Server oryx/127.3.211.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":68,"threads_started":1}
W20260925 15:20:55.570696 13074 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:55.565604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:52067 (local address 127.3.211.254:38693)
0925 15:20:55.565714 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:55.565716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:55.565725 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:20:55.565754 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:55.565755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:55.565771 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:55.565838 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:55.566217 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:55.566717 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:55.567462 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:55.567653 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:55.567722 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:55.570622 (+ 2900us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.211.194:52067: BlockingRecv error: recv got EOF from 127.3.211.194:52067 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 25 15:20:55 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.195@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:55 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.195@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:20:55.708005 13076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:55.702294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38693 (local address 127.3.211.195:52437)
0925 15:20:55.702574 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:55.702589 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:55.702672 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:55.702863 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:55.702864 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:55.703047 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:55.703092 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:55.703094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:55.703913 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:55.703914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:55.704685 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:55.704688 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:55.705205 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:55.705207 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:55.705287 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:55.705906 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:55.705912 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:55.707943 (+ 2031us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.211.254:38693: Server oryx/127.3.211.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":120,"threads_started":1}
W20260925 15:20:55.708196 13074 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:55.702466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:52437 (local address 127.3.211.254:38693)
0925 15:20:55.702578 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:55.702581 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:55.702650 (+ 69us) server_negotiation.cc:407] Connection header received
0925 15:20:55.702742 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:55.702743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:55.702783 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:55.702855 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:55.703176 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:55.703743 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:55.704800 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:55.704997 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:55.705067 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:55.708124 (+ 3057us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.211.195:52437: BlockingRecv error: recv got EOF from 127.3.211.195:52437 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 25 15:20:56 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.193@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:56 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.193@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:20:56.432857 13077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:56.426612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38693 (local address 127.3.211.193:55921)
0925 15:20:56.426839 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:56.426854 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:56.427079 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:56.427356 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:56.427358 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:56.427602 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:56.427656 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:56.427658 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:56.428458 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:56.428463 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:56.429004 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:56.429006 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:56.429464 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:56.429466 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:56.429552 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:56.430269 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:56.430275 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:56.432790 (+ 2515us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.211.254:38693: Server oryx/127.3.211.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":85,"threads_started":1}
W20260925 15:20:56.433143 13078 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:56.426809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:55921 (local address 127.3.211.254:38693)
0925 15:20:56.427058 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:56.427060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:56.427070 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:20:56.427204 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:56.427205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:56.427218 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:56.427297 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:56.427779 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:56.428306 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:56.429097 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:56.429298 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:56.429360 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:56.433065 (+ 3705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.211.193:55921: BlockingRecv error: recv got EOF from 127.3.211.193:55921 (error 108)
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":207,"threads_started":1}
Sep 25 15:20:56 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.194@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:56 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.194@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:20:56.576489 13079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:56.571181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38693 (local address 127.3.211.194:35635)
0925 15:20:56.571325 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:56.571339 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:56.571426 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:56.571615 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:56.571616 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:56.571867 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:56.571927 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:56.571929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:56.572687 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:56.572689 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:56.573421 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:56.573428 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:56.573915 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:56.573916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:56.573982 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:56.574511 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:56.574516 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:56.576424 (+ 1908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.211.254:38693: Server oryx/127.3.211.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":72,"threads_started":1}
W20260925 15:20:56.576694 13078 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:56.571331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:35635 (local address 127.3.211.254:38693)
0925 15:20:56.571414 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:56.571416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:56.571431 (+ 15us) server_negotiation.cc:407] Connection header received
0925 15:20:56.571507 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:56.571508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:56.571522 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:56.571598 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:56.572012 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:56.572586 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:56.573596 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:56.573815 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:56.573886 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:56.576618 (+ 2732us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.211.194:35635: BlockingRecv error: recv got EOF from 127.3.211.194:35635 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 15:20:56 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.195@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:56 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.195@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:20:56.713687 13080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:56.708721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38693 (local address 127.3.211.195:36571)
0925 15:20:56.708901 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:56.708914 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:56.708973 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:56.709116 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:56.709117 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:56.709317 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:56.709367 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:56.709370 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:56.710051 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:56.710052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:56.710623 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:56.710625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:56.711012 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:56.711013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:56.711071 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:56.711598 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:56.711603 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:56.713624 (+ 2021us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.211.254:38693: Server oryx/127.3.211.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":75,"threads_started":1}
W20260925 15:20:56.713943 13078 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:56.708918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:36571 (local address 127.3.211.254:38693)
0925 15:20:56.708968 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:56.708970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:56.708979 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:20:56.709028 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:56.709029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:56.709042 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:56.709102 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:56.709455 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:56.709956 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:56.710725 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:56.710922 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:56.711011 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:56.713895 (+ 2884us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.211.195:36571: BlockingRecv error: recv got EOF from 127.3.211.195:36571 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 25 15:20:57 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.193@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:57 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.193@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:20:57.438769 13081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:57.433708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38693 (local address 127.3.211.193:55375)
0925 15:20:57.433838 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:57.433855 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:57.433912 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:57.434116 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:57.434117 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:57.434300 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:57.434348 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:57.434350 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:57.435131 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:57.435132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:57.435724 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:57.435725 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:57.436247 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:57.436248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:57.436295 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:57.436905 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:57.436910 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:57.438708 (+ 1798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.211.254:38693: Server oryx/127.3.211.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":78,"threads_started":1}
W20260925 15:20:57.438980 13082 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:57.433818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:55375 (local address 127.3.211.254:38693)
0925 15:20:57.433952 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:57.433954 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:57.433962 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:20:57.433986 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:57.433987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:57.434000 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:57.434069 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:57.434473 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:57.435053 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:57.435825 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:57.436165 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:57.436235 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:57.438904 (+ 2669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.211.193:55375: BlockingRecv error: recv got EOF from 127.3.211.193:55375 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
Sep 25 15:20:57 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.194@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:57 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.194@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:20:57.583041 13083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:57.577184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38693 (local address 127.3.211.194:50693)
0925 15:20:57.577408 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:57.577422 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:57.577500 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:57.577659 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:57.577659 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:57.577839 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:57.577914 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:57.577916 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:57.578685 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:57.578687 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:57.579352 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:57.579354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:57.579724 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:57.579725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:57.579776 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:57.580306 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:57.580313 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:57.582973 (+ 2660us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.211.254:38693: Server oryx/127.3.211.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":122,"threads_started":1}
W20260925 15:20:57.583244 13082 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:57.577364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:50693 (local address 127.3.211.254:38693)
0925 15:20:57.577442 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:57.577444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:57.577494 (+ 50us) server_negotiation.cc:407] Connection header received
0925 15:20:57.577560 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:57.577561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:57.577572 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:57.577623 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:57.578023 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:57.578544 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:57.579447 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:57.579645 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:57.579727 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:57.583132 (+ 3405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.211.194:50693: BlockingRecv error: recv got EOF from 127.3.211.194:50693 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 25 15:20:57 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.195@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:57 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.195@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:20:57.719077 13084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:57.714130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38693 (local address 127.3.211.195:42427)
0925 15:20:57.714277 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:57.714291 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:57.714361 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:57.714511 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:57.714512 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:57.714697 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:57.714749 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:57.714751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:57.715448 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:57.715448 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:57.715993 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:57.715995 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:57.716498 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:57.716499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:57.716551 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:57.717054 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:57.717059 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:57.719015 (+ 1956us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.211.254:38693: Server oryx/127.3.211.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":66,"threads_started":1}
W20260925 15:20:57.719422 13082 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:57.714247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:42427 (local address 127.3.211.254:38693)
0925 15:20:57.714323 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:57.714325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:57.714336 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:20:57.714427 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:57.714428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:57.714439 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:57.714488 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:57.714846 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:57.715352 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:57.716089 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:57.716285 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:57.716408 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:57.719321 (+ 2913us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.211.195:42427: BlockingRecv error: recv got EOF from 127.3.211.195:42427 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 25 15:20:58 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.193@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:58 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.193@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:20:58.445658 13085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:58.439483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38693 (local address 127.3.211.193:58895)
0925 15:20:58.439676 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:58.439702 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:58.439847 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:58.440190 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:58.440192 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:58.440469 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:58.440543 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:58.440546 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:58.441446 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:58.441449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:58.442005 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:58.442007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:58.442492 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:58.442492 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:58.442533 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:58.443142 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:58.443146 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:58.445592 (+ 2446us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.211.254:38693: Server oryx/127.3.211.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":63,"threads_started":1}
W20260925 15:20:58.445883 13086 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:58.439683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:58895 (local address 127.3.211.254:38693)
0925 15:20:58.439824 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:58.439826 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:58.439844 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:20:58.439976 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:58.439977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:58.439991 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:58.440080 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:58.440646 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:58.441344 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:58.442106 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:58.442352 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:58.442508 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:58.445791 (+ 3283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.211.193:58895: BlockingRecv error: recv got EOF from 127.3.211.193:58895 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":64,"threads_started":1}
Sep 25 15:20:58 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.194@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:58 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.194@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:20:58.588274 13087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:58.583646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38693 (local address 127.3.211.194:58393)
0925 15:20:58.583830 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:58.583841 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:58.583903 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:58.584021 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:58.584022 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:58.584186 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:58.584236 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:58.584238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:58.584952 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:58.584953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:58.585482 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:58.585484 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:58.585839 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:58.585840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:58.585887 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:58.586450 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:58.586455 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:58.588214 (+ 1759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.211.254:38693: Server oryx/127.3.211.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":61,"threads_started":1}
W20260925 15:20:58.588539 13086 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:58.583797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:58393 (local address 127.3.211.254:38693)
0925 15:20:58.583874 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:58.583880 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:58.583890 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:20:58.583953 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:58.583954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:58.583964 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:58.584013 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:58.584305 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:58.584866 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:58.585562 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:58.585749 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:58.585818 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:58.588481 (+ 2663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.211.194:58393: BlockingRecv error: recv got EOF from 127.3.211.194:58393 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 25 15:20:58 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.195@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:58 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.195@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:20:58.725219 13088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:58.719711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38693 (local address 127.3.211.195:57479)
0925 15:20:58.719868 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:58.719886 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:58.719939 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:58.720110 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:58.720111 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:58.720282 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:58.720400 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:58.720402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:58.721116 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:58.721116 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:58.721648 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:58.721650 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:58.722065 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:58.722066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:58.722117 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:58.722689 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:58.722694 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:58.724964 (+ 2270us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.211.254:38693: Server oryx/127.3.211.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":82,"threads_started":1}
W20260925 15:20:58.725317 13086 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:58.719897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:57479 (local address 127.3.211.254:38693)
0925 15:20:58.719970 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:58.719972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:58.719981 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:20:58.720014 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:58.720014 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:58.720025 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:58.720090 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:58.720536 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:58.721031 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:58.721767 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:58.721960 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:58.722032 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:58.725259 (+ 3227us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.211.195:57479: BlockingRecv error: recv got EOF from 127.3.211.195:57479 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 25 15:20:59 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.193@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:59 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.193@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:20:59.451658 13089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:59.446246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38693 (local address 127.3.211.193:53329)
0925 15:20:59.446399 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:59.446411 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:59.446499 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:59.446968 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:59.446970 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:59.447142 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:59.447196 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:59.447198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:59.448166 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:59.448167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:59.448815 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:59.448817 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:59.449235 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:59.449236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:59.449279 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:59.449826 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:59.449830 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:59.451596 (+ 1766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.211.254:38693: Server oryx/127.3.211.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":87,"threads_started":1}
W20260925 15:20:59.451870 13090 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:59.446502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:53329 (local address 127.3.211.254:38693)
0925 15:20:59.446719 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:59.446721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:59.446735 (+ 14us) server_negotiation.cc:407] Connection header received
0925 15:20:59.446776 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:59.446777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:59.446791 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:59.446901 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:59.447334 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:59.448040 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:59.448932 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:59.449157 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:59.449228 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:59.451796 (+ 2568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.211.193:53329: BlockingRecv error: recv got EOF from 127.3.211.193:53329 (error 108)
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":73,"threads_started":1}
Sep 25 15:20:59 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.194@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:59 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.194@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:20:59.594457 13091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:59.588948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38693 (local address 127.3.211.194:43163)
0925 15:20:59.589132 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:59.589158 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:59.589227 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:59.589477 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:59.589478 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:59.589639 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:59.589688 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:59.589690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:59.590704 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:59.590706 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:59.591484 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:59.591490 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:59.591899 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:59.591900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:59.591961 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:59.592551 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:59.592556 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:59.594394 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.211.254:38693: Server oryx/127.3.211.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":86,"threads_started":1}
W20260925 15:20:59.594684 13090 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:59.589034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:43163 (local address 127.3.211.254:38693)
0925 15:20:59.589151 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:59.589153 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:59.589323 (+ 170us) server_negotiation.cc:407] Connection header received
0925 15:20:59.589358 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:59.589359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:59.589371 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:59.589438 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:59.589801 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:59.590565 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:59.591582 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:59.591795 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:59.591882 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:59.594608 (+ 2726us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.211.194:43163: BlockingRecv error: recv got EOF from 127.3.211.194:43163 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 25 15:20:59 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.195@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:20:59 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.195@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:20:59.731069 13092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:20:59.725859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38693 (local address 127.3.211.195:58821)
0925 15:20:59.725999 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 15:20:59.726018 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:20:59.726090 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:20:59.726426 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:20:59.726428 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:20:59.726635 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:20:59.726691 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:59.726694 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:59.727373 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:59.727374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:59.727940 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:20:59.727941 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:20:59.728299 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:20:59.728300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:20:59.728423 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:59.728924 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:20:59.728929 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:20:59.731003 (+ 2074us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.211.254:38693: Server oryx/127.3.211.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":91,"threads_started":1}
W20260925 15:20:59.731292 13090 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:20:59.725964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:58821 (local address 127.3.211.254:38693)
0925 15:20:59.726019 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 15:20:59.726021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:20:59.726222 (+ 201us) server_negotiation.cc:407] Connection header received
0925 15:20:59.726253 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:20:59.726254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:20:59.726269 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:20:59.726349 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:20:59.726775 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:59.727281 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:59.728031 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:20:59.728214 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:20:59.728270 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:20:59.731206 (+ 2936us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.211.195:58821: BlockingRecv error: recv got EOF from 127.3.211.195:58821 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 25 15:21:00 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.193@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:21:00 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.193@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:21:00.463613 13093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:00.452384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38693 (local address 127.3.211.193:37303)
0925 15:21:00.452551 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:00.452565 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:00.452643 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:00.452843 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:00.452844 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:00.453038 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:21:00.453084 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:00.453085 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:00.453807 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:00.453808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:00.454351 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:00.454353 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:00.454758 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:00.454759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:00.454817 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:00.460986 (+ 6169us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:21:00.460993 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:21:00.463532 (+ 2539us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.211.254:38693: Server oryx/127.3.211.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":81,"threads_started":1}
W20260925 15:21:00.463878 13094 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:21:00.452465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:37303 (local address 127.3.211.254:38693)
0925 15:21:00.452623 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:00.452626 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:00.452642 (+ 16us) server_negotiation.cc:407] Connection header received
0925 15:21:00.452736 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:00.452736 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:00.452747 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:00.452836 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:00.453177 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:00.453693 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:00.454451 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:00.454659 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:00.454749 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:00.463781 (+ 9032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.211.193:37303: BlockingRecv error: recv got EOF from 127.3.211.193:37303 (error 108)
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":85,"threads_started":1}
Sep 25 15:21:00 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.194@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:21:00 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.194@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:21:00.601404 13095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:00.595209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38693 (local address 127.3.211.194:34351)
0925 15:21:00.595451 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:00.595469 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:00.595536 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:00.595745 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:00.595746 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:00.595921 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:21:00.595984 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:00.595986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:00.596832 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:00.596833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:00.597387 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:00.597389 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:00.597775 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:00.597776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:00.597816 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:00.598444 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:21:00.598451 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:21:00.601330 (+ 2879us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.211.254:38693: Server oryx/127.3.211.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":188,"threads_started":1}
W20260925 15:21:00.601545 13094 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:21:00.595351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:34351 (local address 127.3.211.254:38693)
0925 15:21:00.595460 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:00.595462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:00.595569 (+ 107us) server_negotiation.cc:407] Connection header received
0925 15:21:00.595607 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:00.595608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:00.595620 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:00.595687 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:00.596097 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:00.596730 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:00.597494 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:00.597701 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:00.597766 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:00.601492 (+ 3726us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.211.194:34351: BlockingRecv error: recv got EOF from 127.3.211.194:34351 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 25 15:21:00 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.195@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:21:00 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.195@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:21:00.737109 13096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:00.731739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38693 (local address 127.3.211.195:43897)
0925 15:21:00.732009 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:00.732023 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:00.732095 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:00.732256 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:00.732257 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:00.732526 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:21:00.732584 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:00.732587 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:00.733291 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:00.733291 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:00.733896 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:00.733898 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:00.734310 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:00.734311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:00.734359 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:00.734942 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:21:00.734946 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:21:00.737049 (+ 2103us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.211.254:38693: Server oryx/127.3.211.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":180,"threads_started":1}
W20260925 15:21:00.737277 13094 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:21:00.731873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:43897 (local address 127.3.211.254:38693)
0925 15:21:00.731966 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:00.731968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:00.732101 (+ 133us) server_negotiation.cc:407] Connection header received
0925 15:21:00.732150 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:00.732151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:00.732166 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:00.732240 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:00.732664 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:00.733212 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:00.734006 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:00.734197 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:00.734282 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:00.737169 (+ 2887us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.211.195:43897: BlockingRecv error: recv got EOF from 127.3.211.195:43897 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 25 15:21:01 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.193@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:21:01 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.193@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:21:01.469724 13097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:01.464556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38693 (local address 127.3.211.193:55195)
0925 15:21:01.464778 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:01.464791 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:01.464852 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:01.464991 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:01.464992 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:01.465152 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:21:01.465200 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:01.465202 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:01.465889 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:01.465890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:01.466468 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:01.466469 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:01.466873 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:01.466874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:01.466919 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:01.467414 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:21:01.467419 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:21:01.469595 (+ 2176us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.211.254:38693: Server oryx/127.3.211.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":115,"threads_started":1}
W20260925 15:21:01.469919 13098 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:21:01.464704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:55195 (local address 127.3.211.254:38693)
0925 15:21:01.464854 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:01.464856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:01.464886 (+ 30us) server_negotiation.cc:407] Connection header received
0925 15:21:01.464910 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:01.464910 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:01.464920 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:01.464971 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:01.465269 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:01.465791 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:01.466574 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:01.466789 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:01.466853 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:01.469729 (+ 2876us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.211.193:55195: BlockingRecv error: recv got EOF from 127.3.211.193:55195 (error 108)
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":92,"threads_started":1}
Sep 25 15:21:01 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.194@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:21:01 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.194@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:21:01.607227 13099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:01.602114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38693 (local address 127.3.211.194:35283)
0925 15:21:01.602380 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:01.602403 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:01.602482 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:01.602636 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:01.602637 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:01.602794 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:21:01.602843 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:01.602844 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:01.603553 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:01.603554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:01.604265 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:01.604267 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:01.604735 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:01.604736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:01.604783 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:01.605338 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:21:01.605344 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:21:01.607168 (+ 1824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.211.254:38693: Server oryx/127.3.211.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":175,"threads_started":1}
W20260925 15:21:01.607407 13098 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:21:01.602266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:35283 (local address 127.3.211.254:38693)
0925 15:21:01.602337 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:01.602339 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:01.602509 (+ 170us) server_negotiation.cc:407] Connection header received
0925 15:21:01.602545 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:01.602546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:01.602557 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:01.602617 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:01.602913 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:01.603444 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:01.604360 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:01.604632 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:01.604691 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:01.607318 (+ 2627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.211.194:35283: BlockingRecv error: recv got EOF from 127.3.211.194:35283 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 25 15:21:01 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.195@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:21:01 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.195@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:21:01.743301 13100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:01.737847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38693 (local address 127.3.211.195:49017)
0925 15:21:01.738054 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:01.738065 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:01.738158 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:01.738326 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:01.738327 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:01.738569 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:21:01.738629 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:01.738633 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:01.739500 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:01.739501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:01.740073 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:01.740075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:01.740610 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:01.740611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:01.740668 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:01.741241 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:21:01.741246 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:21:01.743228 (+ 1982us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.211.254:38693: Server oryx/127.3.211.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":86,"threads_started":1}
W20260925 15:21:01.743482 13098 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:21:01.738088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:49017 (local address 127.3.211.254:38693)
0925 15:21:01.738162 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:01.738164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:01.738173 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:21:01.738225 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:01.738226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:01.738246 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:01.738318 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:01.738714 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:01.739399 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:01.740204 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:01.740515 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:01.740608 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:01.743416 (+ 2808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.211.195:49017: BlockingRecv error: recv got EOF from 127.3.211.195:49017 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 25 15:21:02 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.193@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:21:02 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.193@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:21:02.476154 13101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:02.470420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38693 (local address 127.3.211.193:44603)
0925 15:21:02.470615 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:02.470626 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:02.470716 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:02.470884 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:02.470885 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:02.471066 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:21:02.471114 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:02.471115 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:02.472141 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:02.472142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:02.472789 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:02.472792 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:02.473277 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:02.473278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:02.473332 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:02.473873 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:21:02.473878 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:21:02.476016 (+ 2138us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.211.254:38693: Server oryx/127.3.211.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":99,"threads_started":1}
W20260925 15:21:02.476202 13102 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:21:02.470473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:44603 (local address 127.3.211.254:38693)
0925 15:21:02.470603 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:02.470605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:02.470695 (+ 90us) server_negotiation.cc:407] Connection header received
0925 15:21:02.470778 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:02.470779 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:02.470789 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:02.470852 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:02.471316 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:02.471979 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:02.472926 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:02.473177 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:02.473254 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:02.476127 (+ 2873us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.211.193:44603: BlockingRecv error: recv got EOF from 127.3.211.193:44603 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":65,"threads_started":1}
Sep 25 15:21:02 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.194@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:21:02 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.194@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:21:02.613786 13103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:02.607713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38693 (local address 127.3.211.194:40003)
0925 15:21:02.607876 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:02.607893 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:02.607949 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:02.608084 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:02.608085 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:02.608251 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:21:02.608299 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:02.608300 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:02.609094 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:02.609096 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:02.609945 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:02.609948 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:02.610368 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:02.610369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:02.610420 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:02.610932 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:21:02.610937 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:21:02.613671 (+ 2734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.211.254:38693: Server oryx/127.3.211.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":109,"threads_started":1}
W20260925 15:21:02.613847 13102 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:21:02.607844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:40003 (local address 127.3.211.254:38693)
0925 15:21:02.607896 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:02.607898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:02.607937 (+ 39us) server_negotiation.cc:407] Connection header received
0925 15:21:02.607996 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:02.607997 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:02.608007 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:02.608066 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:02.608411 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:02.608941 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:02.610060 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:02.610264 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:02.610326 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:02.613779 (+ 3453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.211.194:40003: BlockingRecv error: recv got EOF from 127.3.211.194:40003 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 25 15:21:02 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.195@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:21:02 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.195@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:21:02.749667 13104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:02.743893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38693 (local address 127.3.211.195:56429)
0925 15:21:02.744010 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:02.744024 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:02.744133 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:02.744436 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:02.744437 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:02.744649 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:21:02.744696 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:02.744698 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:02.745403 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:02.745404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:02.745962 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:02.745963 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:02.746343 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:02.746344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:02.746399 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:02.747009 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:21:02.747017 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:21:02.749573 (+ 2556us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.211.254:38693: Server oryx/127.3.211.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":58,"threads_started":1}
W20260925 15:21:02.749862 13102 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:21:02.743999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:56429 (local address 127.3.211.254:38693)
0925 15:21:02.744071 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:02.744073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:02.744109 (+ 36us) server_negotiation.cc:407] Connection header received
0925 15:21:02.744256 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:02.744258 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:02.744275 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:02.744424 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:02.744788 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:02.745293 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:02.746063 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:02.746261 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:02.746344 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:02.749767 (+ 3423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.211.195:56429: BlockingRecv error: recv got EOF from 127.3.211.195:56429 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 25 15:21:03 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.193@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:21:03 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.193@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:21:03.482875 13105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:03.476931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38693 (local address 127.3.211.193:51725)
0925 15:21:03.477058 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:03.477077 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:03.477131 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:03.477549 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:03.477554 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:03.477735 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:21:03.477782 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:03.477784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:03.478956 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:03.478957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:03.479530 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:03.479531 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:03.480075 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:03.480076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:03.480135 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:03.480725 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:21:03.480730 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:21:03.482797 (+ 2067us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.211.254:38693: Server oryx/127.3.211.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":73,"threads_started":1}
W20260925 15:21:03.483002 13106 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:21:03.477165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:51725 (local address 127.3.211.254:38693)
0925 15:21:03.477325 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:03.477327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:03.477337 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:21:03.477364 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:03.477364 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:03.477374 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:03.477476 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:03.478052 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:03.478817 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:03.479694 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:03.479951 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:03.480047 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:03.482922 (+ 2875us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.211.193:51725: BlockingRecv error: recv got EOF from 127.3.211.193:51725 (error 108)
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":90,"threads_started":1}
Sep 25 15:21:03 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.194@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:21:03 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.194@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:21:03.619935 13107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:03.614416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38693 (local address 127.3.211.194:42723)
0925 15:21:03.614567 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:03.614578 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:03.614654 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:03.614823 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:03.614824 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:03.614990 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:21:03.615037 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:03.615038 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:03.615781 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:03.615782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:03.616461 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:03.616464 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:03.617005 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:03.617006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:03.617067 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:03.617792 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:21:03.617804 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:21:03.619872 (+ 2068us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.211.254:38693: Server oryx/127.3.211.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":97,"threads_started":1}
W20260925 15:21:03.620129 13106 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:21:03.614576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:42723 (local address 127.3.211.254:38693)
0925 15:21:03.614643 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:03.614645 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:03.614658 (+ 13us) server_negotiation.cc:407] Connection header received
0925 15:21:03.614700 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:03.614702 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:03.614712 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:03.614770 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:03.615160 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:03.615663 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:03.616572 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:03.616904 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:03.616971 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:03.620023 (+ 3052us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.211.194:42723: BlockingRecv error: recv got EOF from 127.3.211.194:42723 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 25 15:21:03 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.195@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:21:03 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.195@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:21:03.755563 13108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:03.750306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38693 (local address 127.3.211.195:51301)
0925 15:21:03.750477 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:03.750489 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:03.750570 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:03.750762 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:03.750763 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:03.751009 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:21:03.751067 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:03.751070 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:03.751809 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:03.751810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:03.752503 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:03.752505 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:03.752893 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:03.752894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:03.752940 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:03.753466 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:21:03.753471 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:21:03.755443 (+ 1972us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.211.254:38693: Server oryx/127.3.211.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":73,"threads_started":1}
W20260925 15:21:03.755604 13106 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:21:03.750418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:51301 (local address 127.3.211.254:38693)
0925 15:21:03.750478 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:03.750481 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:03.750568 (+ 87us) server_negotiation.cc:407] Connection header received
0925 15:21:03.750626 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:03.750626 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:03.750639 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:03.750731 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:03.751195 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:03.751699 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:03.752603 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:03.752798 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:03.752869 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:03.755538 (+ 2669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.211.195:51301: BlockingRecv error: recv got EOF from 127.3.211.195:51301 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 25 15:21:04 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.193@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:21:04 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.193@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:21:04.490952 13109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:04.483463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38693 (local address 127.3.211.193:33557)
0925 15:21:04.483596 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:04.483608 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:04.483668 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:04.484008 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:04.484010 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:04.484190 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:21:04.484241 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:04.484243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:04.485565 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:04.485567 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:04.486790 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:04.486798 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:04.487379 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:04.487380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:04.487451 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:04.488028 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:21:04.488035 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:21:04.490885 (+ 2850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.211.254:38693: Server oryx/127.3.211.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":85,"threads_started":1}
W20260925 15:21:04.491243 13110 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:21:04.483612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:33557 (local address 127.3.211.254:38693)
0925 15:21:04.483832 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:04.483835 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:04.483847 (+ 12us) server_negotiation.cc:407] Connection header received
0925 15:21:04.483877 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:04.483877 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:04.483896 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:04.483970 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:04.484365 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:04.485411 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:04.486951 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:04.487233 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:04.487327 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:04.491096 (+ 3769us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.211.193:33557: BlockingRecv error: recv got EOF from 127.3.211.193:33557 (error 108)
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":203,"threads_started":1}
Sep 25 15:21:04 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.194@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:21:04 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.194@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:21:04.627027 13111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:04.620713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38693 (local address 127.3.211.194:58321)
0925 15:21:04.620883 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:04.620898 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:04.620964 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:04.621186 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:04.621186 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:04.621448 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:21:04.622047 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:04.622057 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:04.623018 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:04.623020 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:04.623621 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:04.623623 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:04.624092 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:04.624093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:04.624138 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:04.624676 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:21:04.624680 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:21:04.626938 (+ 2258us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.211.254:38693: Server oryx/127.3.211.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":110,"threads_started":1}
W20260925 15:21:04.627189 13110 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:21:04.620926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:58321 (local address 127.3.211.254:38693)
0925 15:21:04.620994 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:04.620996 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:04.621014 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:21:04.621064 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:04.621065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:04.621077 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:04.621156 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:04.622249 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:04.622887 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:04.623760 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:04.623998 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:04.624071 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:04.627101 (+ 3030us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.211.194:58321: BlockingRecv error: recv got EOF from 127.3.211.194:58321 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 25 15:21:04 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.195@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:21:04 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.195@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:21:04.761124 13112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:04.756246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38693 (local address 127.3.211.195:49983)
0925 15:21:04.756419 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:04.756439 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:04.756510 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:04.756698 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:04.756699 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:04.756892 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:21:04.756936 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:04.756937 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:04.757633 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:04.757634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:04.758166 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:04.758168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:04.758528 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:04.758529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:04.758585 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:04.759071 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:21:04.759076 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:21:04.761062 (+ 1986us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.211.254:38693: Server oryx/127.3.211.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":59,"threads_started":1}
W20260925 15:21:04.761329 13110 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:21:04.756407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:49983 (local address 127.3.211.254:38693)
0925 15:21:04.756477 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:04.756479 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:04.756490 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:21:04.756592 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:04.756593 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:04.756609 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:04.756692 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:04.757022 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:04.757546 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:04.758280 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:04.758454 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:04.758531 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:04.761250 (+ 2719us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.211.195:49983: BlockingRecv error: recv got EOF from 127.3.211.195:49983 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 25 15:21:05 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.193@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:21:05 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.193@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:21:05.503322 13113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:05.491623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38693 (local address 127.3.211.193:59679)
0925 15:21:05.491780 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:05.491791 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:05.491844 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:05.492079 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:05.492080 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:05.492260 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:21:05.492306 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:05.492308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:05.493131 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:05.493133 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:05.493703 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:05.493705 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:05.494257 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:05.494258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:05.494315 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:05.501230 (+ 6915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:21:05.501237 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:21:05.503249 (+ 2012us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.211.254:38693: Server oryx/127.3.211.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":90,"threads_started":1}
W20260925 15:21:05.503595 13114 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:21:05.491745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:59679 (local address 127.3.211.254:38693)
0925 15:21:05.491878 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:05.491881 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:05.491890 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:21:05.491925 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:05.491925 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:05.491935 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:05.492012 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:05.492465 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:05.492985 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:05.493862 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:05.494065 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:05.494137 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:05.503510 (+ 9373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.211.193:59679: BlockingRecv error: recv got EOF from 127.3.211.193:59679 (error 108)
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":79,"threads_started":1}
Sep 25 15:21:05 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.194@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:21:05 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.194@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:21:05.634452 13115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:05.627747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38693 (local address 127.3.211.194:40403)
0925 15:21:05.627914 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:05.627925 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:05.627986 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:05.628219 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:05.628220 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:05.628578 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:21:05.628626 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:05.628629 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:05.629389 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:05.629390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:05.630050 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:05.630053 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:05.630773 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:05.630774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:05.630843 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:05.631601 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:21:05.631609 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:21:05.634357 (+ 2748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.211.254:38693: Server oryx/127.3.211.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":116,"threads_started":1}
W20260925 15:21:05.634640 13114 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:21:05.627859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:40403 (local address 127.3.211.254:38693)
0925 15:21:05.627962 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:05.627964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:05.627985 (+ 21us) server_negotiation.cc:407] Connection header received
0925 15:21:05.628056 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:05.628057 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:05.628068 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:05.628148 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:05.628749 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:05.629280 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:05.630209 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:05.630525 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:05.630603 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:05.634564 (+ 3961us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.211.194:40403: BlockingRecv error: recv got EOF from 127.3.211.194:40403 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 25 15:21:05 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.195@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:21:05 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.195@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:21:05.767032 13116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:05.761782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38693 (local address 127.3.211.195:59953)
0925 15:21:05.761978 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:05.761990 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:05.762061 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:05.762244 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:05.762245 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:05.762427 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:21:05.762473 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:05.762474 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:05.763425 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:05.763426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:05.764110 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:05.764113 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:05.764521 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:05.764522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:05.764573 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:05.765081 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:21:05.765085 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:21:05.766961 (+ 1876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.211.254:38693: Server oryx/127.3.211.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":84,"threads_started":1}
W20260925 15:21:05.767159 13114 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:21:05.761857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:59953 (local address 127.3.211.254:38693)
0925 15:21:05.761973 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:05.761975 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:05.762059 (+ 84us) server_negotiation.cc:407] Connection header received
0925 15:21:05.762109 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:05.762110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:05.762122 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:05.762177 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:05.762587 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:05.763261 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:05.764212 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:05.764428 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:05.764488 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:05.767073 (+ 2585us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.211.195:59953: BlockingRecv error: recv got EOF from 127.3.211.195:59953 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 25 15:21:06 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.193@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:21:06 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.193@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:21:06.510437 13117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:06.503994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38693 (local address 127.3.211.193:37023)
0925 15:21:06.504182 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:06.504197 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:06.504266 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:06.504470 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:06.504471 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:06.504631 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:21:06.504680 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:06.504681 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:06.505641 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:06.505642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:06.506241 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:06.506243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:06.506728 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:06.506730 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:06.506804 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:06.507525 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:21:06.507530 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:21:06.510370 (+ 2840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.211.254:38693: Server oryx/127.3.211.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":74,"threads_started":1}
W20260925 15:21:06.510912 13118 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:21:06.504063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:37023 (local address 127.3.211.254:38693)
0925 15:21:06.504243 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:06.504245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:06.504254 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:21:06.504377 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:06.504378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:06.504400 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:06.504464 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:06.504862 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:06.505453 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:06.506382 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:06.506624 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:06.506713 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:06.510820 (+ 4107us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.211.193:37023: BlockingRecv error: recv got EOF from 127.3.211.193:37023 (error 108)
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":115,"threads_started":1}
Sep 25 15:21:06 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.194@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:21:06 dist-test-slave-4bk3 krb5kdc[12059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.211.194@KRBTEST.COM for oryx/127.3.211.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:21:06.640453 13119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:06.635159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38693 (local address 127.3.211.194:40177)
0925 15:21:06.635323 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:06.635333 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:06.635386 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:06.635611 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:06.635613 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:06.635803 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:21:06.635856 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:06.635858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:06.636884 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:06.636886 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:06.637493 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:06.637495 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:06.637864 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:06.637865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:06.637904 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:06.638428 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:21:06.638432 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:21:06.640392 (+ 1960us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.211.254:38693: Server oryx/127.3.211.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":106,"threads_started":1}
W20260925 15:21:06.640604 13118 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:21:06.635291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:40177 (local address 127.3.211.254:38693)
0925 15:21:06.635380 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:06.635382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:06.635441 (+ 59us) server_negotiation.cc:407] Connection header received
0925 15:21:06.635475 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:06.635476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:06.635490 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:06.635548 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:06.635941 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:06.636759 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:06.637587 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:06.637783 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:06.637844 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:06.640525 (+ 2681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.211.194:40177: BlockingRecv error: recv got EOF from 127.3.211.194:40177 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 15:21:06.680574 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 12642
I20260925 15:21:06.685357 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 12794
I20260925 15:21:06.689733 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 12927
I20260925 15:21:06.694010 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 12572
2026-09-25T15:21:06Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17627 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T15:21:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:21:06Z Disabled control of system clock
I20260925 15:21:06.723300 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:37063
--webserver_interface=127.3.211.254
--webserver_port=0
--builtin_ntp_servers=127.3.211.212:37745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.211.254:37063
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 15:21:06.811460 13123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:21:06.811710 13123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:21:06.811733 13123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:21:06.813146 13123 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:21:06.813201 13123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:21:06.813215 13123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:21:06.813227 13123 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:21:06.813239 13123 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:21:06.814703 13123 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:37745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.211.254:37063
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:37063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.211.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:21:06.814962 13123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:21:06.815230 13123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:21:06.820937 13131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:21:06.821132 13123 server_base.cc:1034] running on GCE node
W20260925 15:21:06.821211 13129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:21:06.821211 13128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:21:06.821688 13123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:21:06.822062 13123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:21:06.823256 13123 hybrid_clock.cc:648] HybridClock initialized: now 1790349666823221 us; error 69 us; skew 500 ppm
I20260925 15:21:06.824791 13123 webserver.cc:460] Webserver started at http://127.3.211.254:40481/ using document root <none> and password file <none>
I20260925 15:21:06.825066 13123 fs_manager.cc:359] Metadata directory not provided
I20260925 15:21:06.825143 13123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:21:06.825301 13123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:21:06.826452 13123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/master-0/data/instance:
uuid: "3e3c547e06644fb9ab5d8164233d0a31"
format_stamp: "Formatted at 2026-09-25 15:21:06 on dist-test-slave-4bk3"
server_key: "c204f5b9aae09b9adfc13f2f886b876b"
server_key_iv: "c97b07fe5b7404a3f58a2dc79e3e30b1"
server_key_version: "encryptionkey@0"
I20260925 15:21:06.826856 13123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/master-0/wal/instance:
uuid: "3e3c547e06644fb9ab5d8164233d0a31"
format_stamp: "Formatted at 2026-09-25 15:21:06 on dist-test-slave-4bk3"
server_key: "c204f5b9aae09b9adfc13f2f886b876b"
server_key_iv: "c97b07fe5b7404a3f58a2dc79e3e30b1"
server_key_version: "encryptionkey@0"
I20260925 15:21:06.828990 13123 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260925 15:21:06.829825 13137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:06.830018 13123 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:21:06.830132 13123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/master-0/wal
uuid: "3e3c547e06644fb9ab5d8164233d0a31"
format_stamp: "Formatted at 2026-09-25 15:21:06 on dist-test-slave-4bk3"
server_key: "c204f5b9aae09b9adfc13f2f886b876b"
server_key_iv: "c97b07fe5b7404a3f58a2dc79e3e30b1"
server_key_version: "encryptionkey@0"
I20260925 15:21:06.830219 13123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:21:06.842811 13123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:21:06.843176 13123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:21:06.843340 13123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:21:06.847730 13123 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.254:37063
I20260925 15:21:06.847769 13189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.254:37063 every 8 connection(s)
I20260925 15:21:06.848157 13123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
I20260925 15:21:06.849017 13190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:21:06.849021 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 13123
I20260925 15:21:06.849133 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/master-0/wal/instance
I20260925 15:21:06.849349 3919 external_mini_cluster.cc:1428] Setting key e82edf9380cab1b0f5eb1505a241ad41
I20260925 15:21:06.853227 13190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e3c547e06644fb9ab5d8164233d0a31: Bootstrap starting.
I20260925 15:21:06.854030 13190 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e3c547e06644fb9ab5d8164233d0a31: Neither blocks nor log segments found. Creating new log.
I20260925 15:21:06.854383 13190 log.cc:824] T 00000000000000000000000000000000 P 3e3c547e06644fb9ab5d8164233d0a31: Log is configured to *not* fsync() on all Append() calls
I20260925 15:21:06.855329 13190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e3c547e06644fb9ab5d8164233d0a31: No bootstrap required, opened a new log
I20260925 15:21:06.856442 13193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:06.849985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57584 (local address 127.3.211.254:37063)
0925 15:21:06.850226 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:06.850235 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:06.850272 (+ 37us) server_negotiation.cc:407] Connection header received
0925 15:21:06.850735 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:06.850745 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:06.850838 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:06.851006 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:06.851238 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:06.852656 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:06.853618 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:06.854273 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:06.854418 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:06.855329 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:21:06.855333 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:21:06.855333 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:21:06.855837 (+ 504us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:21:06.855886 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:21:06.855903 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:21:06.855984 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:06.856085 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 15:21:06.856177 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":81,"threads_started":1}
I20260925 15:21:06.858029 13190 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e3c547e06644fb9ab5d8164233d0a31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e3c547e06644fb9ab5d8164233d0a31" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 37063 } }
I20260925 15:21:06.858187 13190 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e3c547e06644fb9ab5d8164233d0a31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:21:06.858210 13190 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e3c547e06644fb9ab5d8164233d0a31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e3c547e06644fb9ab5d8164233d0a31, State: Initialized, Role: FOLLOWER
I20260925 15:21:06.858363 13190 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e3c547e06644fb9ab5d8164233d0a31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e3c547e06644fb9ab5d8164233d0a31" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 37063 } }
I20260925 15:21:06.858470 13190 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e3c547e06644fb9ab5d8164233d0a31 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:21:06.858496 13190 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e3c547e06644fb9ab5d8164233d0a31 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:21:06.858527 13190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e3c547e06644fb9ab5d8164233d0a31 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:21:06.859608 13190 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e3c547e06644fb9ab5d8164233d0a31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e3c547e06644fb9ab5d8164233d0a31" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 37063 } }
I20260925 15:21:06.859766 13190 leader_election.cc:302] T 00000000000000000000000000000000 P 3e3c547e06644fb9ab5d8164233d0a31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e3c547e06644fb9ab5d8164233d0a31; no voters:
I20260925 15:21:06.859979 13190 leader_election.cc:288] T 00000000000000000000000000000000 P 3e3c547e06644fb9ab5d8164233d0a31 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:21:06.860051 13195 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e3c547e06644fb9ab5d8164233d0a31 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:21:06.860291 13190 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e3c547e06644fb9ab5d8164233d0a31 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:21:06.860313 13195 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e3c547e06644fb9ab5d8164233d0a31 [term 1 LEADER]: Becoming Leader. State: Replica: 3e3c547e06644fb9ab5d8164233d0a31, State: Running, Role: LEADER
I20260925 15:21:06.860575 13195 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e3c547e06644fb9ab5d8164233d0a31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e3c547e06644fb9ab5d8164233d0a31" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 37063 } }
I20260925 15:21:06.861025 13196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e3c547e06644fb9ab5d8164233d0a31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e3c547e06644fb9ab5d8164233d0a31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e3c547e06644fb9ab5d8164233d0a31" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 37063 } } }
I20260925 15:21:06.861020 13197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e3c547e06644fb9ab5d8164233d0a31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e3c547e06644fb9ab5d8164233d0a31. Latest consensus state: current_term: 1 leader_uuid: "3e3c547e06644fb9ab5d8164233d0a31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e3c547e06644fb9ab5d8164233d0a31" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 37063 } } }
I20260925 15:21:06.861136 13196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e3c547e06644fb9ab5d8164233d0a31 [sys.catalog]: This master's current role is: LEADER
I20260925 15:21:06.861140 13197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e3c547e06644fb9ab5d8164233d0a31 [sys.catalog]: This master's current role is: LEADER
I20260925 15:21:06.861425 13204 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:21:06.861977 13204 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:21:06.863855 13204 catalog_manager.cc:1347] Generated new cluster ID: 70fef5409e2a41fd8be23e1b662940e3
I20260925 15:21:06.863915 13204 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:21:06.883090 13204 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:21:06.883734 13204 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:21:06.889541 13204 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e3c547e06644fb9ab5d8164233d0a31: Generated new TSK 0
I20260925 15:21:06.889925 13204 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:21:06.893282 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.193:0
--local_ip_for_outbound_sockets=127.3.211.193
--webserver_interface=127.3.211.193
--webserver_port=0
--tserver_master_addrs=127.3.211.254:37063
--builtin_ntp_servers=127.3.211.212:37745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 15:21:06.988451 13214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:21:06.988715 13214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:21:06.988757 13214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:21:06.990097 13214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:21:06.990147 13214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:21:06.990175 13214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.193
I20260925 15:21:06.991910 13214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:37745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.211.193
--webserver_port=0
--tserver_master_addrs=127.3.211.254:37063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.193
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:21:06.992213 13214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:21:06.992565 13214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:21:06.998281 13222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:21:06.998418 13214 server_base.cc:1034] running on GCE node
W20260925 15:21:06.998275 13220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:21:06.998272 13219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:21:06.998952 13214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:21:06.999361 13214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:21:07.000514 13214 hybrid_clock.cc:648] HybridClock initialized: now 1790349667000466 us; error 63 us; skew 500 ppm
I20260925 15:21:07.002234 13214 webserver.cc:460] Webserver started at http://127.3.211.193:38143/ using document root <none> and password file <none>
I20260925 15:21:07.002543 13214 fs_manager.cc:359] Metadata directory not provided
I20260925 15:21:07.002624 13214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:21:07.002775 13214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:21:07.003926 13214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-0/data/instance:
uuid: "13af3747c9864492b6b4ed5f3c41ea6c"
format_stamp: "Formatted at 2026-09-25 15:21:06 on dist-test-slave-4bk3"
server_key: "fed9a6f03bca8e42a04b9ba01bedf435"
server_key_iv: "94801b4675fd9bf30a89694f9c33bf3c"
server_key_version: "encryptionkey@0"
I20260925 15:21:07.004307 13214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-0/wal/instance:
uuid: "13af3747c9864492b6b4ed5f3c41ea6c"
format_stamp: "Formatted at 2026-09-25 15:21:06 on dist-test-slave-4bk3"
server_key: "fed9a6f03bca8e42a04b9ba01bedf435"
server_key_iv: "94801b4675fd9bf30a89694f9c33bf3c"
server_key_version: "encryptionkey@0"
I20260925 15:21:07.005932 13214 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:21:07.006878 13228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:07.007354 13214 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:21:07.007592 13214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-0/wal
uuid: "13af3747c9864492b6b4ed5f3c41ea6c"
format_stamp: "Formatted at 2026-09-25 15:21:06 on dist-test-slave-4bk3"
server_key: "fed9a6f03bca8e42a04b9ba01bedf435"
server_key_iv: "94801b4675fd9bf30a89694f9c33bf3c"
server_key_version: "encryptionkey@0"
I20260925 15:21:07.007728 13214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:21:07.031543 13214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:21:07.031924 13214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:21:07.032099 13214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:21:07.032423 13214 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:21:07.032845 13214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:21:07.032879 13214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:07.032949 13214 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:21:07.032975 13214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:07.039139 13214 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.193:34175
I20260925 15:21:07.039235 13341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.193:34175 every 8 connection(s)
I20260925 15:21:07.039595 13214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:21:07.039834 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 13214
I20260925 15:21:07.039955 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-0/wal/instance
I20260925 15:21:07.040185 3919 external_mini_cluster.cc:1428] Setting key d4f38cda11e0a4688a61b18a31c7de1f
I20260925 15:21:07.041649 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.194:0
--local_ip_for_outbound_sockets=127.3.211.194
--webserver_interface=127.3.211.194
--webserver_port=0
--tserver_master_addrs=127.3.211.254:37063
--builtin_ntp_servers=127.3.211.212:37745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 15:21:07.047096 13193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:07.040633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:50821 (local address 127.3.211.254:37063)
0925 15:21:07.040692 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:07.040695 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:07.040951 (+ 256us) server_negotiation.cc:407] Connection header received
0925 15:21:07.042041 (+ 1090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:07.042042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:07.042059 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:07.042119 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:07.043541 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:07.044060 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:07.045345 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:07.045635 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:07.045726 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:07.046764 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:21:07.046766 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:21:07.046766 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:21:07.046772 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:21:07.046789 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:21:07.046793 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:21:07.046847 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:07.046955 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 15:21:07.047005 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 15:21:07.047325 13344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:07.040511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:37063 (local address 127.3.211.193:50821)
0925 15:21:07.040806 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:07.040851 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:07.041923 (+ 1072us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:07.042200 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:07.042204 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:07.042215 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:21:07.043308 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:07.043324 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:07.044297 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:07.044299 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:07.045207 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:07.045211 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:07.045746 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:07.045747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:07.045934 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:07.046572 (+ 638us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:21:07.046573 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:21:07.046606 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:21:07.046634 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:21:07.046856 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:21:07.046861 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:21:07.046864 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:21:07.046959 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 15:21:07.047075 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":111,"threads_started":1}
I20260925 15:21:07.048094 13342 heartbeater.cc:347] Connected to a master server at 127.3.211.254:37063
I20260925 15:21:07.048187 13342 heartbeater.cc:464] Registering TS with master...
I20260925 15:21:07.048476 13342 heartbeater.cc:511] Master 127.3.211.254:37063 requested a full tablet report, sending...
I20260925 15:21:07.049139 13154 ts_manager.cc:194] Registered new tserver with Master: 13af3747c9864492b6b4ed5f3c41ea6c (127.3.211.193:34175)
I20260925 15:21:07.049801 13154 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.211.193:50821
W20260925 15:21:07.130079 13345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:21:07.130393 13345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:21:07.130434 13345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:21:07.131812 13345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:21:07.131897 13345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:21:07.131942 13345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.194
I20260925 15:21:07.133623 13345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:37745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.211.194
--webserver_port=0
--tserver_master_addrs=127.3.211.254:37063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.194
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:21:07.133945 13345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:21:07.134320 13345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:21:07.139997 13351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:21:07.140103 13353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:21:07.140046 13345 server_base.cc:1034] running on GCE node
W20260925 15:21:07.139997 13350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:21:07.140514 13345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:21:07.140810 13345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:21:07.142028 13345 hybrid_clock.cc:648] HybridClock initialized: now 1790349667142000 us; error 38 us; skew 500 ppm
I20260925 15:21:07.144034 13345 webserver.cc:460] Webserver started at http://127.3.211.194:39691/ using document root <none> and password file <none>
I20260925 15:21:07.144379 13345 fs_manager.cc:359] Metadata directory not provided
I20260925 15:21:07.144472 13345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:21:07.144644 13345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:21:07.145821 13345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-1/data/instance:
uuid: "f95ebe0a84ee4b6482a80c4d18f1cc25"
format_stamp: "Formatted at 2026-09-25 15:21:07 on dist-test-slave-4bk3"
server_key: "19b7ab78a07b31a41641619fa9972af8"
server_key_iv: "987687180533eb233cdd198afffd9f91"
server_key_version: "encryptionkey@0"
I20260925 15:21:07.146212 13345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-1/wal/instance:
uuid: "f95ebe0a84ee4b6482a80c4d18f1cc25"
format_stamp: "Formatted at 2026-09-25 15:21:07 on dist-test-slave-4bk3"
server_key: "19b7ab78a07b31a41641619fa9972af8"
server_key_iv: "987687180533eb233cdd198afffd9f91"
server_key_version: "encryptionkey@0"
I20260925 15:21:07.147607 13345 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:21:07.148475 13359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:07.148703 13345 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:21:07.148847 13345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-1/wal
uuid: "f95ebe0a84ee4b6482a80c4d18f1cc25"
format_stamp: "Formatted at 2026-09-25 15:21:07 on dist-test-slave-4bk3"
server_key: "19b7ab78a07b31a41641619fa9972af8"
server_key_iv: "987687180533eb233cdd198afffd9f91"
server_key_version: "encryptionkey@0"
I20260925 15:21:07.148947 13345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:21:07.173331 13345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:21:07.173979 13345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:21:07.174197 13345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:21:07.174467 13345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:21:07.174891 13345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:21:07.174957 13345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:07.174999 13345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:21:07.175029 13345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:07.182111 13345 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.194:37181
I20260925 15:21:07.182173 13472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.194:37181 every 8 connection(s)
I20260925 15:21:07.182659 13345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
I20260925 15:21:07.188247 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 13345
I20260925 15:21:07.188388 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-1/wal/instance
I20260925 15:21:07.188621 3919 external_mini_cluster.cc:1428] Setting key 339d81528a511b8e3c6b4bb583bd00d2
I20260925 15:21:07.189694 13193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:07.183867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:39047 (local address 127.3.211.254:37063)
0925 15:21:07.183940 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:07.183950 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:07.184345 (+ 395us) server_negotiation.cc:407] Connection header received
0925 15:21:07.185017 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:07.185019 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:07.185036 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:07.185111 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:07.185928 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:07.186594 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:07.187747 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:07.187984 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:07.188052 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:07.189338 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:21:07.189339 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:21:07.189339 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:21:07.189346 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:21:07.189362 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:21:07.189367 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:21:07.189425 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:07.189539 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 15:21:07.189578 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 15:21:07.189738 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.195:0
--local_ip_for_outbound_sockets=127.3.211.195
--webserver_interface=127.3.211.195
--webserver_port=0
--tserver_master_addrs=127.3.211.254:37063
--builtin_ntp_servers=127.3.211.212:37745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 15:21:07.189867 13475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:07.183764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:37063 (local address 127.3.211.194:39047)
0925 15:21:07.184117 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:07.184179 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:07.184855 (+ 676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:07.185238 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:07.185243 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:07.185252 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:21:07.185793 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:07.185796 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:07.186732 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:07.186733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:07.187613 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:07.187616 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:07.188103 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:07.188105 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:07.188310 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:07.189104 (+ 794us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:21:07.189104 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:21:07.189137 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:21:07.189163 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:21:07.189440 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:21:07.189446 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:21:07.189449 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:21:07.189519 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 15:21:07.189615 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":131,"threads_started":1}
I20260925 15:21:07.190609 13473 heartbeater.cc:347] Connected to a master server at 127.3.211.254:37063
I20260925 15:21:07.190760 13473 heartbeater.cc:464] Registering TS with master...
I20260925 15:21:07.190966 13473 heartbeater.cc:511] Master 127.3.211.254:37063 requested a full tablet report, sending...
I20260925 15:21:07.191403 13154 ts_manager.cc:194] Registered new tserver with Master: f95ebe0a84ee4b6482a80c4d18f1cc25 (127.3.211.194:37181)
I20260925 15:21:07.191807 13154 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.211.194:39047
W20260925 15:21:07.277176 13476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:21:07.277467 13476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:21:07.277508 13476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:21:07.278777 13476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:21:07.278870 13476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:21:07.278918 13476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.195
I20260925 15:21:07.280512 13476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:37745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.211.195
--webserver_port=0
--tserver_master_addrs=127.3.211.254:37063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.195
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:21:07.280822 13476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:21:07.281174 13476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:21:07.286653 13484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:21:07.286773 13481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:21:07.286653 13482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:21:07.286928 13476 server_base.cc:1034] running on GCE node
I20260925 15:21:07.287249 13476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:21:07.287485 13476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:21:07.288640 13476 hybrid_clock.cc:648] HybridClock initialized: now 1790349667288594 us; error 69 us; skew 500 ppm
I20260925 15:21:07.290064 13476 webserver.cc:460] Webserver started at http://127.3.211.195:40325/ using document root <none> and password file <none>
I20260925 15:21:07.290361 13476 fs_manager.cc:359] Metadata directory not provided
I20260925 15:21:07.290414 13476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:21:07.290524 13476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:21:07.291561 13476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-2/data/instance:
uuid: "e7f1291e7b584e29bfb8023f9b4e6237"
format_stamp: "Formatted at 2026-09-25 15:21:07 on dist-test-slave-4bk3"
server_key: "055e8e84fa847a3cfee3fc45b3fdda97"
server_key_iv: "2f3bd3202e57451135d9c791e50388cf"
server_key_version: "encryptionkey@0"
I20260925 15:21:07.291879 13476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-2/wal/instance:
uuid: "e7f1291e7b584e29bfb8023f9b4e6237"
format_stamp: "Formatted at 2026-09-25 15:21:07 on dist-test-slave-4bk3"
server_key: "055e8e84fa847a3cfee3fc45b3fdda97"
server_key_iv: "2f3bd3202e57451135d9c791e50388cf"
server_key_version: "encryptionkey@0"
I20260925 15:21:07.293263 13476 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:21:07.294154 13490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:07.294381 13476 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:21:07.294502 13476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-2/wal
uuid: "e7f1291e7b584e29bfb8023f9b4e6237"
format_stamp: "Formatted at 2026-09-25 15:21:07 on dist-test-slave-4bk3"
server_key: "055e8e84fa847a3cfee3fc45b3fdda97"
server_key_iv: "2f3bd3202e57451135d9c791e50388cf"
server_key_version: "encryptionkey@0"
I20260925 15:21:07.294585 13476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:21:07.303062 13476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:21:07.303434 13476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:21:07.303597 13476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:21:07.303862 13476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:21:07.304232 13476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:21:07.304291 13476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:07.304363 13476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:21:07.304388 13476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:07.311218 13476 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.195:38063
I20260925 15:21:07.311265 13603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.195:38063 every 8 connection(s)
I20260925 15:21:07.311694 13476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
I20260925 15:21:07.316198 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 13476
I20260925 15:21:07.316372 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-2/wal/instance
I20260925 15:21:07.316573 3919 external_mini_cluster.cc:1428] Setting key 2f74a4aed0ae5016d4c9d66f99d7f0bd
I20260925 15:21:07.317809 13193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:07.312496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:40833 (local address 127.3.211.254:37063)
0925 15:21:07.312564 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:07.312566 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:07.312928 (+ 362us) server_negotiation.cc:407] Connection header received
0925 15:21:07.313513 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:07.313514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:07.313528 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:07.313600 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:07.314323 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:07.314905 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:07.315979 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:07.316181 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:07.316266 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:07.317500 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:21:07.317502 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:21:07.317502 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:21:07.317508 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:21:07.317525 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:21:07.317530 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:21:07.317584 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:07.317704 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 15:21:07.317734 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 15:21:07.318097 13606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:07.312515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:37063 (local address 127.3.211.195:40833)
0925 15:21:07.312751 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:07.312808 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:07.313373 (+ 565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:07.313677 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:07.313681 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:07.313692 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:21:07.314221 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:07.314225 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:07.315008 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:07.315009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:07.315838 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:07.315840 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:07.316275 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:07.316277 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:07.316502 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:07.317269 (+ 767us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:21:07.317270 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:21:07.317315 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:21:07.317353 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:21:07.317606 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:21:07.317612 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:21:07.317615 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:21:07.317696 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 15:21:07.317825 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":100,"threads_started":1}
I20260925 15:21:07.318588 13604 heartbeater.cc:347] Connected to a master server at 127.3.211.254:37063
I20260925 15:21:07.318691 13604 heartbeater.cc:464] Registering TS with master...
I20260925 15:21:07.318902 13604 heartbeater.cc:511] Master 127.3.211.254:37063 requested a full tablet report, sending...
I20260925 15:21:07.319263 13154 ts_manager.cc:194] Registered new tserver with Master: e7f1291e7b584e29bfb8023f9b4e6237 (127.3.211.195:38063)
I20260925 15:21:07.319612 13154 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.211.195:40833
I20260925 15:21:07.329586 3919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:21:07.335145 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 13214
W20260925 15:21:07.335201 13193 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:21:07.333516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57588 (local address 127.3.211.254:37063)
0925 15:21:07.333574 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:07.333576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:07.334589 (+ 1013us) server_negotiation.cc:407] Connection header received
0925 15:21:07.334739 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:07.334763 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:07.334778 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:07.334857 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:07.335105 (+ 248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57588: BlockingRecv error: recv got EOF from 127.0.0.1:57588 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 15:21:07.341992 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 13345
I20260925 15:21:07.347255 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 13476
I20260925 15:21:07.353112 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 13123
2026-09-25T15:21:07Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (660 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T15:21:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:21:07Z Disabled control of system clock
I20260925 15:21:07.383030 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:42331
--webserver_interface=127.3.211.254
--webserver_port=0
--builtin_ntp_servers=127.3.211.212:40681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.211.254:42331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 15:21:07.466368 13615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:21:07.466609 13615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:21:07.466632 13615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:21:07.468030 13615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:21:07.468086 13615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:21:07.468101 13615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:21:07.468113 13615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:21:07.468127 13615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:21:07.469630 13615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:40681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.211.254:42331
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:42331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.211.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:21:07.469933 13615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:21:07.470261 13615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:21:07.475457 13623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:21:07.475438 13621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:21:07.475438 13620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:21:07.475471 13615 server_base.cc:1034] running on GCE node
I20260925 15:21:07.476015 13615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:21:07.476414 13615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:21:07.477613 13615 hybrid_clock.cc:648] HybridClock initialized: now 1790349667477563 us; error 72 us; skew 500 ppm
I20260925 15:21:07.478930 13615 webserver.cc:460] Webserver started at http://127.3.211.254:35243/ using document root <none> and password file <none>
I20260925 15:21:07.479194 13615 fs_manager.cc:359] Metadata directory not provided
I20260925 15:21:07.479269 13615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:21:07.479734 13615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:21:07.481245 13615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/master-0/data/instance:
uuid: "d1035c213ab6492fb104973a34465478"
format_stamp: "Formatted at 2026-09-25 15:21:07 on dist-test-slave-4bk3"
server_key: "2b681a790168f25a3399049a546293cd"
server_key_iv: "f98b559b8f1945ce0a735ac87674fe76"
server_key_version: "encryptionkey@0"
I20260925 15:21:07.481652 13615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/master-0/wal/instance:
uuid: "d1035c213ab6492fb104973a34465478"
format_stamp: "Formatted at 2026-09-25 15:21:07 on dist-test-slave-4bk3"
server_key: "2b681a790168f25a3399049a546293cd"
server_key_iv: "f98b559b8f1945ce0a735ac87674fe76"
server_key_version: "encryptionkey@0"
I20260925 15:21:07.483382 13615 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:21:07.485183 13629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:07.485497 13615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:21:07.485631 13615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/master-0/wal
uuid: "d1035c213ab6492fb104973a34465478"
format_stamp: "Formatted at 2026-09-25 15:21:07 on dist-test-slave-4bk3"
server_key: "2b681a790168f25a3399049a546293cd"
server_key_iv: "f98b559b8f1945ce0a735ac87674fe76"
server_key_version: "encryptionkey@0"
I20260925 15:21:07.485724 13615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:21:07.490042 13615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:21:07.490552 13615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:21:07.490710 13615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:21:07.494822 13615 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.254:42331
I20260925 15:21:07.494871 13681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.254:42331 every 8 connection(s)
I20260925 15:21:07.495244 13615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
I20260925 15:21:07.495973 13682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:21:07.498011 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 13615
I20260925 15:21:07.498134 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/master-0/wal/instance
I20260925 15:21:07.498344 3919 external_mini_cluster.cc:1428] Setting key 014230532b42d87019b32eb07e48b9e7
I20260925 15:21:07.499233 13682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1035c213ab6492fb104973a34465478: Bootstrap starting.
I20260925 15:21:07.499979 13682 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1035c213ab6492fb104973a34465478: Neither blocks nor log segments found. Creating new log.
I20260925 15:21:07.500357 13682 log.cc:824] T 00000000000000000000000000000000 P d1035c213ab6492fb104973a34465478: Log is configured to *not* fsync() on all Append() calls
I20260925 15:21:07.501186 13682 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1035c213ab6492fb104973a34465478: No bootstrap required, opened a new log
I20260925 15:21:07.501416 13685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:07.498949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42652 (local address 127.3.211.254:42331)
0925 15:21:07.499170 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:07.499179 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:07.499210 (+ 31us) server_negotiation.cc:407] Connection header received
0925 15:21:07.499813 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:07.499822 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:07.499879 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:07.500055 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:07.500913 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:21:07.500916 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:21:07.500917 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:21:07.500930 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:21:07.500970 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:21:07.500981 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:21:07.501056 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:07.501132 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 15:21:07.501217 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":81,"threads_started":1}
I20260925 15:21:07.503166 13682 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1035c213ab6492fb104973a34465478 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1035c213ab6492fb104973a34465478" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 42331 } }
I20260925 15:21:07.503334 13682 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1035c213ab6492fb104973a34465478 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:21:07.503382 13682 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1035c213ab6492fb104973a34465478 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1035c213ab6492fb104973a34465478, State: Initialized, Role: FOLLOWER
I20260925 15:21:07.503509 13682 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1035c213ab6492fb104973a34465478 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1035c213ab6492fb104973a34465478" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 42331 } }
I20260925 15:21:07.503600 13682 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1035c213ab6492fb104973a34465478 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:21:07.503645 13682 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1035c213ab6492fb104973a34465478 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:21:07.503693 13682 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1035c213ab6492fb104973a34465478 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:21:07.504259 13682 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1035c213ab6492fb104973a34465478 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1035c213ab6492fb104973a34465478" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 42331 } }
I20260925 15:21:07.504406 13682 leader_election.cc:302] T 00000000000000000000000000000000 P d1035c213ab6492fb104973a34465478 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1035c213ab6492fb104973a34465478; no voters:
I20260925 15:21:07.504598 13682 leader_election.cc:288] T 00000000000000000000000000000000 P d1035c213ab6492fb104973a34465478 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:21:07.504689 13687 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1035c213ab6492fb104973a34465478 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:21:07.504838 13687 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1035c213ab6492fb104973a34465478 [term 1 LEADER]: Becoming Leader. State: Replica: d1035c213ab6492fb104973a34465478, State: Running, Role: LEADER
I20260925 15:21:07.504878 13682 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1035c213ab6492fb104973a34465478 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:21:07.504951 13687 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1035c213ab6492fb104973a34465478 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1035c213ab6492fb104973a34465478" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 42331 } }
I20260925 15:21:07.505604 13689 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1035c213ab6492fb104973a34465478 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1035c213ab6492fb104973a34465478. Latest consensus state: current_term: 1 leader_uuid: "d1035c213ab6492fb104973a34465478" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1035c213ab6492fb104973a34465478" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 42331 } } }
I20260925 15:21:07.505750 13689 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1035c213ab6492fb104973a34465478 [sys.catalog]: This master's current role is: LEADER
I20260925 15:21:07.505614 13688 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1035c213ab6492fb104973a34465478 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1035c213ab6492fb104973a34465478" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1035c213ab6492fb104973a34465478" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 42331 } } }
I20260925 15:21:07.506202 13694 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:21:07.506790 13694 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:21:07.507483 13688 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1035c213ab6492fb104973a34465478 [sys.catalog]: This master's current role is: LEADER
I20260925 15:21:07.508635 13694 catalog_manager.cc:1347] Generated new cluster ID: 71cdb358a6c142a0a45658d475376d62
I20260925 15:21:07.508692 13694 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:21:07.567310 13694 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:21:07.567551 13694 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:21:07.574036 13694 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1035c213ab6492fb104973a34465478: Generated new TSK 0
I20260925 15:21:07.574263 13694 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:21:07.586413 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.193:0
--local_ip_for_outbound_sockets=127.3.211.193
--webserver_interface=127.3.211.193
--webserver_port=0
--tserver_master_addrs=127.3.211.254:42331
--builtin_ntp_servers=127.3.211.212:40681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 15:21:07.680274 13706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:21:07.680557 13706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:21:07.680583 13706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:21:07.681895 13706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:21:07.681945 13706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:21:07.681975 13706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.193
I20260925 15:21:07.683390 13706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:40681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.211.193
--webserver_port=0
--tserver_master_addrs=127.3.211.254:42331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.193
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:21:07.683637 13706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:21:07.683914 13706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:21:07.689323 13712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:21:07.689312 13711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:21:07.689312 13714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:21:07.689649 13706 server_base.cc:1034] running on GCE node
I20260925 15:21:07.689888 13706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:21:07.690160 13706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:21:07.691322 13706 hybrid_clock.cc:648] HybridClock initialized: now 1790349667691310 us; error 29 us; skew 500 ppm
I20260925 15:21:07.692775 13706 webserver.cc:460] Webserver started at http://127.3.211.193:36965/ using document root <none> and password file <none>
I20260925 15:21:07.693054 13706 fs_manager.cc:359] Metadata directory not provided
I20260925 15:21:07.693113 13706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:21:07.693250 13706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:21:07.694408 13706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-0/data/instance:
uuid: "fffb1ff7026a4a6db83e2621d2a2ce56"
format_stamp: "Formatted at 2026-09-25 15:21:07 on dist-test-slave-4bk3"
server_key: "75740b6b919ae790818802b73db40def"
server_key_iv: "2d8f92f040470b4142a7c8ad62f9fb86"
server_key_version: "encryptionkey@0"
I20260925 15:21:07.694787 13706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-0/wal/instance:
uuid: "fffb1ff7026a4a6db83e2621d2a2ce56"
format_stamp: "Formatted at 2026-09-25 15:21:07 on dist-test-slave-4bk3"
server_key: "75740b6b919ae790818802b73db40def"
server_key_iv: "2d8f92f040470b4142a7c8ad62f9fb86"
server_key_version: "encryptionkey@0"
I20260925 15:21:07.696522 13706 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:21:07.697528 13720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:07.697773 13706 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:21:07.697860 13706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-0/wal
uuid: "fffb1ff7026a4a6db83e2621d2a2ce56"
format_stamp: "Formatted at 2026-09-25 15:21:07 on dist-test-slave-4bk3"
server_key: "75740b6b919ae790818802b73db40def"
server_key_iv: "2d8f92f040470b4142a7c8ad62f9fb86"
server_key_version: "encryptionkey@0"
I20260925 15:21:07.697965 13706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:21:07.701947 13706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:21:07.702366 13706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:21:07.702481 13706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:21:07.702642 13706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:21:07.702970 13706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:21:07.703029 13706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:07.703070 13706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:21:07.703101 13706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:07.709843 13706 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.193:33953
I20260925 15:21:07.709942 13833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.193:33953 every 8 connection(s)
I20260925 15:21:07.710312 13706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:21:07.712586 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 13706
I20260925 15:21:07.712702 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-0/wal/instance
I20260925 15:21:07.712967 3919 external_mini_cluster.cc:1428] Setting key 5f5e2141bbb0cdbaaba2289d179e27c5
I20260925 15:21:07.713987 13685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:07.711206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:52113 (local address 127.3.211.254:42331)
0925 15:21:07.711268 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:07.711270 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:07.711568 (+ 298us) server_negotiation.cc:407] Connection header received
0925 15:21:07.712012 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:07.712014 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:07.712021 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:07.712098 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:07.713688 (+ 1590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:21:07.713689 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:21:07.713690 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:21:07.713696 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:21:07.713713 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:21:07.713717 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:21:07.713780 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:07.713891 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 15:21:07.713909 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 15:21:07.714066 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.194:0
--local_ip_for_outbound_sockets=127.3.211.194
--webserver_interface=127.3.211.194
--webserver_port=0
--tserver_master_addrs=127.3.211.254:42331
--builtin_ntp_servers=127.3.211.212:40681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 15:21:07.714252 13836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:07.711169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:42331 (local address 127.3.211.193:52113)
0925 15:21:07.711362 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:07.711405 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:07.711879 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:07.712174 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:07.712178 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:07.712186 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:21:07.713472 (+ 1286us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:21:07.713473 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:21:07.713508 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:21:07.713542 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:21:07.713786 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:21:07.713792 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:21:07.713796 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:21:07.713906 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 15:21:07.713989 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":85,"threads_started":1}
I20260925 15:21:07.715695 13834 heartbeater.cc:347] Connected to a master server at 127.3.211.254:42331
I20260925 15:21:07.715986 13834 heartbeater.cc:464] Registering TS with master...
I20260925 15:21:07.716401 13834 heartbeater.cc:511] Master 127.3.211.254:42331 requested a full tablet report, sending...
I20260925 15:21:07.717092 13646 ts_manager.cc:194] Registered new tserver with Master: fffb1ff7026a4a6db83e2621d2a2ce56 (127.3.211.193:33953)
W20260925 15:21:07.802459 13837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:21:07.802706 13837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:21:07.802732 13837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:21:07.804158 13837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:21:07.804225 13837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:21:07.804265 13837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.194
I20260925 15:21:07.806003 13837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:40681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.211.194
--webserver_port=0
--tserver_master_addrs=127.3.211.254:42331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.194
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:21:07.806375 13837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:21:07.806728 13837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:21:07.812124 13843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:21:07.812182 13837 server_base.cc:1034] running on GCE node
W20260925 15:21:07.812119 13842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:21:07.812124 13845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:21:07.812569 13837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:21:07.812855 13837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:21:07.814014 13837 hybrid_clock.cc:648] HybridClock initialized: now 1790349667813989 us; error 37 us; skew 500 ppm
I20260925 15:21:07.815434 13837 webserver.cc:460] Webserver started at http://127.3.211.194:39277/ using document root <none> and password file <none>
I20260925 15:21:07.815718 13837 fs_manager.cc:359] Metadata directory not provided
I20260925 15:21:07.815794 13837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:21:07.815953 13837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:21:07.817132 13837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-1/data/instance:
uuid: "a2f2f9dc727e4b639f1592ae7d2f22ae"
format_stamp: "Formatted at 2026-09-25 15:21:07 on dist-test-slave-4bk3"
server_key: "630325039970718a1f853cba2dbc9269"
server_key_iv: "b49627b9640ab0991dd80299eeb797d8"
server_key_version: "encryptionkey@0"
I20260925 15:21:07.817533 13837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-1/wal/instance:
uuid: "a2f2f9dc727e4b639f1592ae7d2f22ae"
format_stamp: "Formatted at 2026-09-25 15:21:07 on dist-test-slave-4bk3"
server_key: "630325039970718a1f853cba2dbc9269"
server_key_iv: "b49627b9640ab0991dd80299eeb797d8"
server_key_version: "encryptionkey@0"
I20260925 15:21:07.818964 13837 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:21:07.819906 13851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:07.820139 13837 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:21:07.820245 13837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-1/wal
uuid: "a2f2f9dc727e4b639f1592ae7d2f22ae"
format_stamp: "Formatted at 2026-09-25 15:21:07 on dist-test-slave-4bk3"
server_key: "630325039970718a1f853cba2dbc9269"
server_key_iv: "b49627b9640ab0991dd80299eeb797d8"
server_key_version: "encryptionkey@0"
I20260925 15:21:07.820384 13837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:21:07.824014 13837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:21:07.824318 13837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:21:07.824509 13837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:21:07.824738 13837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:21:07.825086 13837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:21:07.825143 13837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:07.825186 13837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:21:07.825212 13837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:07.831797 13837 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.194:34499
I20260925 15:21:07.831837 13964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.194:34499 every 8 connection(s)
I20260925 15:21:07.832206 13837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
I20260925 15:21:07.835480 13685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:07.832951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:34289 (local address 127.3.211.254:42331)
0925 15:21:07.833000 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:07.833002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:07.833281 (+ 279us) server_negotiation.cc:407] Connection header received
0925 15:21:07.833813 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:07.833814 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:07.833828 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:07.833900 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:07.835194 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:21:07.835195 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:21:07.835196 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:21:07.835201 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:21:07.835217 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:21:07.835222 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:21:07.835283 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:07.835385 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 15:21:07.835401 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 15:21:07.835716 13967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:07.832942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:42331 (local address 127.3.211.194:34289)
0925 15:21:07.833153 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:07.833189 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:07.833695 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:07.833981 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:07.833986 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:07.833993 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:21:07.835025 (+ 1032us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:21:07.835025 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:21:07.835056 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:21:07.835088 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:21:07.835283 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:21:07.835296 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:21:07.835300 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:21:07.835397 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 15:21:07.835466 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":96,"threads_started":1}
I20260925 15:21:07.836547 13965 heartbeater.cc:347] Connected to a master server at 127.3.211.254:42331
I20260925 15:21:07.836652 13965 heartbeater.cc:464] Registering TS with master...
I20260925 15:21:07.836827 13965 heartbeater.cc:511] Master 127.3.211.254:42331 requested a full tablet report, sending...
I20260925 15:21:07.837316 13646 ts_manager.cc:194] Registered new tserver with Master: a2f2f9dc727e4b639f1592ae7d2f22ae (127.3.211.194:34499)
I20260925 15:21:07.840425 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 13837
I20260925 15:21:07.840512 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-1/wal/instance
I20260925 15:21:07.840719 3919 external_mini_cluster.cc:1428] Setting key 49290f29b35a5ba035af16900796b843
I20260925 15:21:07.841661 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.195:0
--local_ip_for_outbound_sockets=127.3.211.195
--webserver_interface=127.3.211.195
--webserver_port=0
--tserver_master_addrs=127.3.211.254:42331
--builtin_ntp_servers=127.3.211.212:40681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 15:21:07.926911 13968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:21:07.927121 13968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:21:07.927146 13968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:21:07.928457 13968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:21:07.928506 13968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:21:07.928535 13968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.195
I20260925 15:21:07.929979 13968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:40681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.211.195
--webserver_port=0
--tserver_master_addrs=127.3.211.254:42331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.195
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:21:07.930265 13968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:21:07.930608 13968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:21:07.935869 13976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:21:07.935912 13973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:21:07.935916 13974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:21:07.936092 13968 server_base.cc:1034] running on GCE node
I20260925 15:21:07.936400 13968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:21:07.936688 13968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:21:07.937867 13968 hybrid_clock.cc:648] HybridClock initialized: now 1790349667937846 us; error 29 us; skew 500 ppm
I20260925 15:21:07.939591 13968 webserver.cc:460] Webserver started at http://127.3.211.195:45021/ using document root <none> and password file <none>
I20260925 15:21:07.939906 13968 fs_manager.cc:359] Metadata directory not provided
I20260925 15:21:07.939986 13968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:21:07.940142 13968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:21:07.941315 13968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-2/data/instance:
uuid: "a948fecff6dc414dbd37b5716f77fefa"
format_stamp: "Formatted at 2026-09-25 15:21:07 on dist-test-slave-4bk3"
server_key: "69379247cdebf3b87f2109dfc40ba0dc"
server_key_iv: "e47ceda4e5d1358d1b92faf58f564059"
server_key_version: "encryptionkey@0"
I20260925 15:21:07.941687 13968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-2/wal/instance:
uuid: "a948fecff6dc414dbd37b5716f77fefa"
format_stamp: "Formatted at 2026-09-25 15:21:07 on dist-test-slave-4bk3"
server_key: "69379247cdebf3b87f2109dfc40ba0dc"
server_key_iv: "e47ceda4e5d1358d1b92faf58f564059"
server_key_version: "encryptionkey@0"
I20260925 15:21:07.943089 13968 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 15:21:07.943929 13982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:07.944119 13968 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:21:07.944221 13968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-2/wal
uuid: "a948fecff6dc414dbd37b5716f77fefa"
format_stamp: "Formatted at 2026-09-25 15:21:07 on dist-test-slave-4bk3"
server_key: "69379247cdebf3b87f2109dfc40ba0dc"
server_key_iv: "e47ceda4e5d1358d1b92faf58f564059"
server_key_version: "encryptionkey@0"
I20260925 15:21:07.944314 13968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:21:07.947844 13968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:21:07.948135 13968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:21:07.948287 13968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:21:07.948526 13968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:21:07.948850 13968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:21:07.948904 13968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:07.948945 13968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:21:07.948973 13968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:07.954926 13968 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.195:35149
I20260925 15:21:07.954962 14095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.195:35149 every 8 connection(s)
I20260925 15:21:07.955331 13968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
I20260925 15:21:07.956696 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 13968
I20260925 15:21:07.956795 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790349610595785-3919-0/minicluster-data/ts-2/wal/instance
I20260925 15:21:07.956975 3919 external_mini_cluster.cc:1428] Setting key 431db86de7c1d992550b23f5ee218af6
I20260925 15:21:07.958694 13685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:07.956205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:40259 (local address 127.3.211.254:42331)
0925 15:21:07.956275 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:07.956278 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:07.956583 (+ 305us) server_negotiation.cc:407] Connection header received
0925 15:21:07.957138 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:07.957139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:07.957146 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:07.957223 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:07.958493 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:21:07.958494 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:21:07.958495 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:21:07.958498 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:21:07.958512 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:21:07.958515 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:21:07.958563 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:07.958631 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 15:21:07.958643 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 15:21:07.958930 14098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:07.956204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:42331 (local address 127.3.211.195:40259)
0925 15:21:07.956452 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:07.956493 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:07.957014 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:07.957280 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:07.957284 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:07.957297 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:21:07.958351 (+ 1054us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:21:07.958351 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:21:07.958379 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:21:07.958408 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:21:07.958575 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:21:07.958579 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:21:07.958582 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:21:07.958641 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 15:21:07.958689 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":95,"threads_started":1}
I20260925 15:21:07.959350 14096 heartbeater.cc:347] Connected to a master server at 127.3.211.254:42331
I20260925 15:21:07.959463 14096 heartbeater.cc:464] Registering TS with master...
I20260925 15:21:07.959612 14096 heartbeater.cc:511] Master 127.3.211.254:42331 requested a full tablet report, sending...
I20260925 15:21:07.959965 13646 ts_manager.cc:194] Registered new tserver with Master: a948fecff6dc414dbd37b5716f77fefa (127.3.211.195:35149)
I20260925 15:21:07.968982 3919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:21:07.970880 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 13706
W20260925 15:21:07.970888 13685 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:21:07.970355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42654 (local address 127.3.211.254:42331)
0925 15:21:07.970436 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:07.970438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:07.970508 (+ 70us) server_negotiation.cc:407] Connection header received
0925 15:21:07.970542 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:07.970543 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:07.970548 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:07.970597 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:07.970791 (+ 194us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42654: BlockingRecv error: recv got EOF from 127.0.0.1:42654 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 15:21:07.975332 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 13837
I20260925 15:21:07.980239 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 13968
I20260925 15:21:07.984294 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 13615
2026-09-25T15:21:07Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (628 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:21:08 dist-test-slave-4bk3 krb5kdc[14106](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:21:08 dist-test-slave-4bk3 krb5kdc[14106](info): set up 2 sockets
Sep 25 15:21:08 dist-test-slave-4bk3 krb5kdc[14106](info): commencing operation
krb5kdc: starting...
W20260925 15:21:09.045296 3919 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Sep 25 15:21:09 dist-test-slave-4bk3 krb5kdc[14106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:21:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:21:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.211.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
I20260925 15:21:09.175490 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:39343
--webserver_interface=127.3.211.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:41953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.211.254:39343
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:21:09.260016 14119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:21:09.260250 14119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:21:09.260282 14119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:21:09.261658 14119 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:21:09.261715 14119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:21:09.261730 14119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:21:09.261749 14119 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:21:09.261761 14119 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:21:09.263414 14119 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:41953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.211.254:39343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:39343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.211.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:21:09.263717 14119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:21:09.264002 14119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:21:09.269264 14127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:21:09.269382 14119 server_base.cc:1034] running on GCE node
W20260925 15:21:09.269282 14124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:21:09.269479 14125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:21:09.269788 14119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:21:09.270124 14119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:21:09.271311 14119 hybrid_clock.cc:648] HybridClock initialized: now 1790349669271300 us; error 32 us; skew 500 ppm
Sep 25 15:21:09 dist-test-slave-4bk3 krb5kdc[14106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349669, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:21:09.273970 14119 init.cc:377] Logged in from keytab as kudu/127.3.211.254@KRBTEST.COM (short username kudu)
I20260925 15:21:09.274488 14119 webserver.cc:460] Webserver started at http://127.3.211.254:46853/ using document root <none> and password file <none>
I20260925 15:21:09.274730 14119 fs_manager.cc:359] Metadata directory not provided
I20260925 15:21:09.274804 14119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:21:09.274966 14119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:21:09.276120 14119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/master-0/data/instance:
uuid: "6f13fd2c0b8542cfabcf036f7fbf272b"
format_stamp: "Formatted at 2026-09-25 15:21:09 on dist-test-slave-4bk3"
server_key: "557a80ef3214f620763051715d4b3308"
server_key_iv: "66056a856e0aaa745cf8aec75d6949e8"
server_key_version: "encryptionkey@0"
I20260925 15:21:09.276547 14119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/master-0/wal/instance:
uuid: "6f13fd2c0b8542cfabcf036f7fbf272b"
format_stamp: "Formatted at 2026-09-25 15:21:09 on dist-test-slave-4bk3"
server_key: "557a80ef3214f620763051715d4b3308"
server_key_iv: "66056a856e0aaa745cf8aec75d6949e8"
server_key_version: "encryptionkey@0"
I20260925 15:21:09.278036 14119 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:21:09.278887 14134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:09.279093 14119 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 15:21:09.279196 14119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/master-0/wal
uuid: "6f13fd2c0b8542cfabcf036f7fbf272b"
format_stamp: "Formatted at 2026-09-25 15:21:09 on dist-test-slave-4bk3"
server_key: "557a80ef3214f620763051715d4b3308"
server_key_iv: "66056a856e0aaa745cf8aec75d6949e8"
server_key_version: "encryptionkey@0"
I20260925 15:21:09.279282 14119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:21:09.300161 14119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:21:09.303380 14119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:21:09.303596 14119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:21:09.308035 14119 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.254:39343
I20260925 15:21:09.308082 14186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.254:39343 every 8 connection(s)
I20260925 15:21:09.308547 14119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
I20260925 15:21:09.309643 14187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:21:09.310766 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 14119
I20260925 15:21:09.310885 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/master-0/wal/instance
I20260925 15:21:09.311089 3919 external_mini_cluster.cc:1428] Setting key 7f50aac5183edc0a5c1a7b5b77611922
I20260925 15:21:09.312999 14187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f13fd2c0b8542cfabcf036f7fbf272b: Bootstrap starting.
I20260925 15:21:09.313860 14187 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f13fd2c0b8542cfabcf036f7fbf272b: Neither blocks nor log segments found. Creating new log.
I20260925 15:21:09.314188 14187 log.cc:824] T 00000000000000000000000000000000 P 6f13fd2c0b8542cfabcf036f7fbf272b: Log is configured to *not* fsync() on all Append() calls
I20260925 15:21:09.315014 14187 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f13fd2c0b8542cfabcf036f7fbf272b: No bootstrap required, opened a new log
I20260925 15:21:09.317502 14187 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f13fd2c0b8542cfabcf036f7fbf272b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f13fd2c0b8542cfabcf036f7fbf272b" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 39343 } }
I20260925 15:21:09.317701 14187 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f13fd2c0b8542cfabcf036f7fbf272b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:21:09.317744 14187 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f13fd2c0b8542cfabcf036f7fbf272b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f13fd2c0b8542cfabcf036f7fbf272b, State: Initialized, Role: FOLLOWER
I20260925 15:21:09.317904 14187 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f13fd2c0b8542cfabcf036f7fbf272b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f13fd2c0b8542cfabcf036f7fbf272b" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 39343 } }
I20260925 15:21:09.318011 14187 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f13fd2c0b8542cfabcf036f7fbf272b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:21:09.318055 14187 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f13fd2c0b8542cfabcf036f7fbf272b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:21:09.318102 14187 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f13fd2c0b8542cfabcf036f7fbf272b [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:21:09.318852 14187 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f13fd2c0b8542cfabcf036f7fbf272b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f13fd2c0b8542cfabcf036f7fbf272b" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 39343 } }
I20260925 15:21:09.318948 14187 leader_election.cc:302] T 00000000000000000000000000000000 P 6f13fd2c0b8542cfabcf036f7fbf272b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f13fd2c0b8542cfabcf036f7fbf272b; no voters:
Sep 25 15:21:09 dist-test-slave-4bk3 krb5kdc[14106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:21:09.319155 14187 leader_election.cc:288] T 00000000000000000000000000000000 P 6f13fd2c0b8542cfabcf036f7fbf272b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:21:09.319175 14192 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f13fd2c0b8542cfabcf036f7fbf272b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:21:09.319312 14192 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f13fd2c0b8542cfabcf036f7fbf272b [term 1 LEADER]: Becoming Leader. State: Replica: 6f13fd2c0b8542cfabcf036f7fbf272b, State: Running, Role: LEADER
I20260925 15:21:09.319407 14192 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f13fd2c0b8542cfabcf036f7fbf272b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f13fd2c0b8542cfabcf036f7fbf272b" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 39343 } }
I20260925 15:21:09.319463 14187 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f13fd2c0b8542cfabcf036f7fbf272b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:21:09.319737 14193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f13fd2c0b8542cfabcf036f7fbf272b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f13fd2c0b8542cfabcf036f7fbf272b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f13fd2c0b8542cfabcf036f7fbf272b" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 39343 } } }
I20260925 15:21:09.319833 14193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f13fd2c0b8542cfabcf036f7fbf272b [sys.catalog]: This master's current role is: LEADER
I20260925 15:21:09.320124 14198 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:21:09.320370 14194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f13fd2c0b8542cfabcf036f7fbf272b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f13fd2c0b8542cfabcf036f7fbf272b. Latest consensus state: current_term: 1 leader_uuid: "6f13fd2c0b8542cfabcf036f7fbf272b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f13fd2c0b8542cfabcf036f7fbf272b" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 39343 } } }
I20260925 15:21:09.320447 14194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f13fd2c0b8542cfabcf036f7fbf272b [sys.catalog]: This master's current role is: LEADER
I20260925 15:21:09.320700 14198 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:21:09.322552 14198 catalog_manager.cc:1347] Generated new cluster ID: e7346128e2524ef4a81d36417ccb197b
I20260925 15:21:09.322607 14198 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:21:09.323685 14190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:09.311779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47272 (local address 127.3.211.254:39343)
0925 15:21:09.311999 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:09.312016 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:09.312046 (+ 30us) server_negotiation.cc:407] Connection header received
0925 15:21:09.312705 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:09.312713 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:09.312765 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:09.312899 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:09.313787 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:09.314995 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:09.316072 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:09.316638 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:09.316734 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:09.319723 (+ 2989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:21:09.319728 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:21:09.319729 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:21:09.319743 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:21:09.321959 (+ 2216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:09.322398 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:09.322402 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:09.322403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:09.322522 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:09.322754 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:09.322755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:09.322756 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:09.323052 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:21:09.323190 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:09.323365 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 15:21:09.323495 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":120,"threads_started":1}
I20260925 15:21:09.333847 14198 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:21:09.334638 14198 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:21:09.344638 14198 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f13fd2c0b8542cfabcf036f7fbf272b: Generated new TSK 0
I20260925 15:21:09.344913 14198 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.211.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab.
Entry for principal kudu/127.3.211.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab.
I20260925 15:21:09.380180 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.193:0
--local_ip_for_outbound_sockets=127.3.211.193
--webserver_interface=127.3.211.193
--webserver_port=0
--tserver_master_addrs=127.3.211.254:39343
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab
--principal=kudu/127.3.211.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:41953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:21:09.469413 14213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:21:09.469636 14213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:21:09.469663 14213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:21:09.470932 14213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:21:09.470984 14213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:21:09.471014 14213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.193
I20260925 15:21:09.472651 14213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:41953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab
--principal=kudu/127.3.211.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.211.193
--webserver_port=0
--tserver_master_addrs=127.3.211.254:39343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.193
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:21:09.472939 14213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:21:09.473194 14213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:21:09.478884 14219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:21:09.478892 14221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:21:09.479198 14218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:21:09.479198 14213 server_base.cc:1034] running on GCE node
I20260925 15:21:09.479640 14213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:21:09.479970 14213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:21:09.481153 14213 hybrid_clock.cc:648] HybridClock initialized: now 1790349669481134 us; error 47 us; skew 500 ppm
Sep 25 15:21:09 dist-test-slave-4bk3 krb5kdc[14106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349669, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:21:09.484268 14213 init.cc:377] Logged in from keytab as kudu/127.3.211.193@KRBTEST.COM (short username kudu)
I20260925 15:21:09.484905 14213 webserver.cc:460] Webserver started at http://127.3.211.193:36191/ using document root <none> and password file <none>
I20260925 15:21:09.485180 14213 fs_manager.cc:359] Metadata directory not provided
I20260925 15:21:09.485234 14213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:21:09.485376 14213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:21:09.486542 14213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/ts-0/data/instance:
uuid: "613fa2f8468045a8a59bf14e27a951d2"
format_stamp: "Formatted at 2026-09-25 15:21:09 on dist-test-slave-4bk3"
server_key: "2afe7c1d7bceea54b6837517e70ab73e"
server_key_iv: "70a886c7379ee1cb9705d995ebfac026"
server_key_version: "encryptionkey@0"
I20260925 15:21:09.486929 14213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/ts-0/wal/instance:
uuid: "613fa2f8468045a8a59bf14e27a951d2"
format_stamp: "Formatted at 2026-09-25 15:21:09 on dist-test-slave-4bk3"
server_key: "2afe7c1d7bceea54b6837517e70ab73e"
server_key_iv: "70a886c7379ee1cb9705d995ebfac026"
server_key_version: "encryptionkey@0"
I20260925 15:21:09.488524 14213 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:21:09.489586 14228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:09.490083 14213 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:21:09.490247 14213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/ts-0/wal
uuid: "613fa2f8468045a8a59bf14e27a951d2"
format_stamp: "Formatted at 2026-09-25 15:21:09 on dist-test-slave-4bk3"
server_key: "2afe7c1d7bceea54b6837517e70ab73e"
server_key_iv: "70a886c7379ee1cb9705d995ebfac026"
server_key_version: "encryptionkey@0"
I20260925 15:21:09.490351 14213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:21:09.523197 14213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:21:09.526063 14213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:21:09.526257 14213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:21:09.526507 14213 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:21:09.526857 14213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:21:09.526918 14213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:09.526954 14213 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:21:09.526983 14213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:09.533368 14213 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.193:37585
I20260925 15:21:09.533428 14341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.193:37585 every 8 connection(s)
I20260925 15:21:09.533789 14213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:21:09.536806 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 14213
I20260925 15:21:09.536922 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/ts-0/wal/instance
I20260925 15:21:09.537086 3919 external_mini_cluster.cc:1428] Setting key 00d4563751e4c07e9ca95f3dcd209d14
Sep 25 15:21:09 dist-test-slave-4bk3 krb5kdc[14106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349669, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.193@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:21:09.544749 14190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:09.534698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:46355 (local address 127.3.211.254:39343)
0925 15:21:09.534769 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:09.534770 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:09.534968 (+ 198us) server_negotiation.cc:407] Connection header received
0925 15:21:09.535501 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:09.535502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:09.535513 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:09.535580 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:09.536672 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:09.537318 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:09.538407 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:09.538635 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:09.538717 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:09.541197 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:21:09.541198 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:21:09.541199 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:21:09.541207 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:21:09.542651 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:09.543114 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:09.543115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:09.543115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:09.543180 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:09.543456 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:09.543457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:09.543458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:09.543588 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:21:09.543668 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:09.544307 (+ 639us) server_negotiation.cc:299] Negotiation successful
0925 15:21:09.544513 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 15:21:09.544997 14344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:09.534625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:39343 (local address 127.3.211.193:46355)
0925 15:21:09.534808 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:09.534852 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:09.535382 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:09.535636 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:09.535640 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:09.536244 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:21:09.536539 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:09.536545 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:09.537443 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:09.537445 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:09.538294 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:09.538297 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:09.538740 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:09.538741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:09.538879 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:09.539576 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:21:09.539585 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:21:09.541082 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:21:09.542765 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:21:09.542774 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:21:09.542776 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:21:09.543032 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:21:09.543252 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:21:09.543253 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:21:09.543253 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:21:09.543385 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:21:09.543658 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:21:09.543665 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:21:09.543822 (+ 157us) client_negotiation.cc:769] Sending connection context
0925 15:21:09.544259 (+ 437us) client_negotiation.cc:240] Negotiation successful
0925 15:21:09.544700 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":85,"threads_started":1}
WARNING: no policy specified for kudu/127.3.211.194@KRBTEST.COM; defaulting to no policy
I20260925 15:21:09.545887 14342 heartbeater.cc:347] Connected to a master server at 127.3.211.254:39343
I20260925 15:21:09.546020 14342 heartbeater.cc:464] Registering TS with master...
I20260925 15:21:09.546232 14342 heartbeater.cc:511] Master 127.3.211.254:39343 requested a full tablet report, sending...
I20260925 15:21:09.546880 14151 ts_manager.cc:194] Registered new tserver with Master: 613fa2f8468045a8a59bf14e27a951d2 (127.3.211.193:37585)
I20260925 15:21:09.547552 14151 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.193@KRBTEST.COM'} at 127.3.211.193:46355
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab.
Entry for principal kudu/127.3.211.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab.
I20260925 15:21:09.567201 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.194:0
--local_ip_for_outbound_sockets=127.3.211.194
--webserver_interface=127.3.211.194
--webserver_port=0
--tserver_master_addrs=127.3.211.254:39343
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab
--principal=kudu/127.3.211.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:41953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:21:09.653833 14347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:21:09.654093 14347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:21:09.654122 14347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:21:09.655357 14347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:21:09.655407 14347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:21:09.655437 14347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.194
I20260925 15:21:09.657039 14347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:41953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab
--principal=kudu/127.3.211.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.211.194
--webserver_port=0
--tserver_master_addrs=127.3.211.254:39343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.194
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:21:09.657264 14347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:21:09.657465 14347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:21:09.662895 14353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:21:09.663008 14355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:21:09.663076 14347 server_base.cc:1034] running on GCE node
W20260925 15:21:09.663030 14352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:21:09.663368 14347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:21:09.663582 14347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:21:09.664744 14347 hybrid_clock.cc:648] HybridClock initialized: now 1790349669664723 us; error 34 us; skew 500 ppm
Sep 25 15:21:09 dist-test-slave-4bk3 krb5kdc[14106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349669, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:21:09.668143 14347 init.cc:377] Logged in from keytab as kudu/127.3.211.194@KRBTEST.COM (short username kudu)
I20260925 15:21:09.668771 14347 webserver.cc:460] Webserver started at http://127.3.211.194:42617/ using document root <none> and password file <none>
I20260925 15:21:09.669044 14347 fs_manager.cc:359] Metadata directory not provided
I20260925 15:21:09.669123 14347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:21:09.669401 14347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:21:09.670588 14347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/ts-1/data/instance:
uuid: "6d4d3c4f4e1441d78cf4fd6e4e9eafed"
format_stamp: "Formatted at 2026-09-25 15:21:09 on dist-test-slave-4bk3"
server_key: "9d242f4527ae4dfe72032614793eaef9"
server_key_iv: "d29ac260d7856d4d060e2ed77dabeb8e"
server_key_version: "encryptionkey@0"
I20260925 15:21:09.671000 14347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/ts-1/wal/instance:
uuid: "6d4d3c4f4e1441d78cf4fd6e4e9eafed"
format_stamp: "Formatted at 2026-09-25 15:21:09 on dist-test-slave-4bk3"
server_key: "9d242f4527ae4dfe72032614793eaef9"
server_key_iv: "d29ac260d7856d4d060e2ed77dabeb8e"
server_key_version: "encryptionkey@0"
I20260925 15:21:09.672727 14347 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260925 15:21:09.673712 14362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:09.673939 14347 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:21:09.674052 14347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/ts-1/wal
uuid: "6d4d3c4f4e1441d78cf4fd6e4e9eafed"
format_stamp: "Formatted at 2026-09-25 15:21:09 on dist-test-slave-4bk3"
server_key: "9d242f4527ae4dfe72032614793eaef9"
server_key_iv: "d29ac260d7856d4d060e2ed77dabeb8e"
server_key_version: "encryptionkey@0"
I20260925 15:21:09.674137 14347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:21:09.698616 14347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:21:09.701056 14347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:21:09.701247 14347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:21:09.701514 14347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:21:09.701874 14347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:21:09.701936 14347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:09.701983 14347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:21:09.702013 14347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:09.709442 14347 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.194:46777
I20260925 15:21:09.709482 14475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.194:46777 every 8 connection(s)
I20260925 15:21:09.709838 14347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
I20260925 15:21:09.712970 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 14347
I20260925 15:21:09.713106 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/ts-1/wal/instance
I20260925 15:21:09.713289 3919 external_mini_cluster.cc:1428] Setting key b70e056f0d8467d458290c3e531484d3
Sep 25 15:21:09 dist-test-slave-4bk3 krb5kdc[14106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349669, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.194@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:21:09.721103 14190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:09.710903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:46757 (local address 127.3.211.254:39343)
0925 15:21:09.711011 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:09.711014 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:09.711187 (+ 173us) server_negotiation.cc:407] Connection header received
0925 15:21:09.711690 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:09.711691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:09.711705 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:09.711779 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:09.712919 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:09.713590 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:09.715130 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:09.715368 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:09.715453 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:09.718094 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:21:09.718096 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:21:09.718096 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:21:09.718104 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:21:09.719543 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:09.719967 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:09.719969 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:09.719969 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:09.720055 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:09.720422 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:09.720423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:09.720423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:09.720549 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:21:09.720630 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:09.720916 (+ 286us) server_negotiation.cc:299] Negotiation successful
0925 15:21:09.721005 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 15:21:09.721352 14478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:09.710748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:39343 (local address 127.3.211.194:46757)
0925 15:21:09.711036 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:09.711082 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:09.711598 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:09.711836 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:09.711840 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:09.712531 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:21:09.712770 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:09.712778 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:09.713697 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:09.713698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:09.715020 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:09.715024 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:09.715478 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:09.715479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:09.715634 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:09.716232 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:21:09.716240 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:21:09.717995 (+ 1755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:21:09.719629 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:21:09.719636 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:21:09.719639 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:21:09.719878 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:21:09.720139 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:21:09.720140 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:21:09.720141 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:21:09.720301 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:21:09.720620 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:21:09.720626 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:21:09.720775 (+ 149us) client_negotiation.cc:769] Sending connection context
0925 15:21:09.720877 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 15:21:09.721053 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":103,"threads_started":1}
WARNING: no policy specified for kudu/127.3.211.195@KRBTEST.COM; defaulting to no policy
I20260925 15:21:09.721990 14476 heartbeater.cc:347] Connected to a master server at 127.3.211.254:39343
I20260925 15:21:09.722108 14476 heartbeater.cc:464] Registering TS with master...
I20260925 15:21:09.722323 14476 heartbeater.cc:511] Master 127.3.211.254:39343 requested a full tablet report, sending...
I20260925 15:21:09.722714 14151 ts_manager.cc:194] Registered new tserver with Master: 6d4d3c4f4e1441d78cf4fd6e4e9eafed (127.3.211.194:46777)
I20260925 15:21:09.723134 14151 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.194@KRBTEST.COM'} at 127.3.211.194:46757
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab.
Entry for principal kudu/127.3.211.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab.
I20260925 15:21:09.743436 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.195:0
--local_ip_for_outbound_sockets=127.3.211.195
--webserver_interface=127.3.211.195
--webserver_port=0
--tserver_master_addrs=127.3.211.254:39343
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab
--principal=kudu/127.3.211.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:41953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:21:09.828994 14481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:21:09.829244 14481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:21:09.829273 14481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:21:09.830681 14481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:21:09.830744 14481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:21:09.830776 14481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.195
I20260925 15:21:09.832551 14481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:41953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab
--principal=kudu/127.3.211.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.211.195
--webserver_port=0
--tserver_master_addrs=127.3.211.254:39343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.195
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:21:09.832837 14481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:21:09.833115 14481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:21:09.838436 14489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:21:09.838488 14481 server_base.cc:1034] running on GCE node
W20260925 15:21:09.838387 14486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:21:09.838404 14487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:21:09.838826 14481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:21:09.839110 14481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:21:09.840286 14481 hybrid_clock.cc:648] HybridClock initialized: now 1790349669840268 us; error 41 us; skew 500 ppm
Sep 25 15:21:09 dist-test-slave-4bk3 krb5kdc[14106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349669, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:21:09.842947 14481 init.cc:377] Logged in from keytab as kudu/127.3.211.195@KRBTEST.COM (short username kudu)
I20260925 15:21:09.843411 14481 webserver.cc:460] Webserver started at http://127.3.211.195:34915/ using document root <none> and password file <none>
I20260925 15:21:09.843696 14481 fs_manager.cc:359] Metadata directory not provided
I20260925 15:21:09.843765 14481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:21:09.843928 14481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:21:09.845289 14481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/ts-2/data/instance:
uuid: "79ae111fcd28421da92d4bec7c94b17a"
format_stamp: "Formatted at 2026-09-25 15:21:09 on dist-test-slave-4bk3"
server_key: "9a8d20efb946c1964b91e1ae542d47ad"
server_key_iv: "05c672e63e55c9af4c9fe3b83986dd9a"
server_key_version: "encryptionkey@0"
I20260925 15:21:09.845685 14481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/ts-2/wal/instance:
uuid: "79ae111fcd28421da92d4bec7c94b17a"
format_stamp: "Formatted at 2026-09-25 15:21:09 on dist-test-slave-4bk3"
server_key: "9a8d20efb946c1964b91e1ae542d47ad"
server_key_iv: "05c672e63e55c9af4c9fe3b83986dd9a"
server_key_version: "encryptionkey@0"
I20260925 15:21:09.847081 14481 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 15:21:09.847971 14496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:09.848212 14481 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:21:09.848361 14481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/ts-2/wal
uuid: "79ae111fcd28421da92d4bec7c94b17a"
format_stamp: "Formatted at 2026-09-25 15:21:09 on dist-test-slave-4bk3"
server_key: "9a8d20efb946c1964b91e1ae542d47ad"
server_key_iv: "05c672e63e55c9af4c9fe3b83986dd9a"
server_key_version: "encryptionkey@0"
I20260925 15:21:09.848464 14481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:21:09.865458 14481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:21:09.868007 14481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:21:09.868191 14481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:21:09.868484 14481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:21:09.868818 14481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:21:09.868876 14481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:09.868922 14481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:21:09.868955 14481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:09.875249 14481 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.195:38649
I20260925 15:21:09.875288 14609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.195:38649 every 8 connection(s)
I20260925 15:21:09.875697 14481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
I20260925 15:21:09.879231 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 14481
I20260925 15:21:09.879366 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/ts-2/wal/instance
I20260925 15:21:09.879572 3919 external_mini_cluster.cc:1428] Setting key b0a70ac5936cebbc61bbcb847e076d87
Sep 25 15:21:09 dist-test-slave-4bk3 krb5kdc[14106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349669, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.195@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:21:09.886193 14190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:09.876609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:44183 (local address 127.3.211.254:39343)
0925 15:21:09.876692 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:09.876695 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:09.876928 (+ 233us) server_negotiation.cc:407] Connection header received
0925 15:21:09.877486 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:09.877487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:09.877498 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:09.877561 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:09.878659 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:09.879238 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:09.880820 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:09.881020 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:09.881091 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:09.883282 (+ 2191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:21:09.883283 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:21:09.883284 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:21:09.883291 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:21:09.884795 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:09.885223 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:09.885225 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:09.885225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:09.885278 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:09.885500 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:09.885500 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:09.885501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:09.885657 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:21:09.885711 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:09.886025 (+ 314us) server_negotiation.cc:299] Negotiation successful
0925 15:21:09.886102 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 15:21:09.886303 14612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:09.876575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:39343 (local address 127.3.211.195:44183)
0925 15:21:09.876746 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:09.876817 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:09.877389 (+ 572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:09.877617 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:09.877620 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:09.878265 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:21:09.878516 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:09.878522 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:09.879425 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:09.879427 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:09.880716 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:09.880720 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:09.881106 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:09.881106 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:09.881208 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:09.881713 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:21:09.881722 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:21:09.883184 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:21:09.884889 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:21:09.884895 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:21:09.884897 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:21:09.885150 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:21:09.885337 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:21:09.885338 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:21:09.885338 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:21:09.885445 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:21:09.885722 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:21:09.885727 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:21:09.885866 (+ 139us) client_negotiation.cc:769] Sending connection context
0925 15:21:09.885967 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 15:21:09.886111 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":96,"threads_started":1}
I20260925 15:21:09.886998 14610 heartbeater.cc:347] Connected to a master server at 127.3.211.254:39343
I20260925 15:21:09.887104 14610 heartbeater.cc:464] Registering TS with master...
I20260925 15:21:09.887454 14610 heartbeater.cc:511] Master 127.3.211.254:39343 requested a full tablet report, sending...
I20260925 15:21:09.887996 14151 ts_manager.cc:194] Registered new tserver with Master: 79ae111fcd28421da92d4bec7c94b17a (127.3.211.195:38649)
I20260925 15:21:09.888576 14151 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.195@KRBTEST.COM'} at 127.3.211.195:44183
I20260925 15:21:09.891753 3919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:21:09.900063 14190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:09.893312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47278 (local address 127.3.211.254:39343)
0925 15:21:09.893378 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:09.893380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:09.893390 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:21:09.893431 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:09.893432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:09.893440 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:09.893489 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:09.894172 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:09.894670 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:09.895650 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:09.895847 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:09.895927 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:09.897357 (+ 1430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:21:09.897358 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:21:09.897358 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:21:09.897366 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:21:09.898818 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:09.899238 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:09.899239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:09.899239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:09.899300 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:09.899495 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:09.899495 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:09.899496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:09.899655 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:21:09.899735 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:09.899925 (+ 190us) server_negotiation.cc:299] Negotiation successful
0925 15:21:09.899995 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 15:21:09.902139 14151 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47278:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:21:09.903777 14151 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:21:09.911473 14624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:09.906875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.195:38649 (local address 127.0.0.1:32792)
0925 15:21:09.906975 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:09.906984 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:09.907028 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:09.907359 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:09.907362 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:09.907368 (+ 6us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:21:09.907458 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:09.907461 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:09.908943 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:09.908944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:09.910157 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:09.910160 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:09.911162 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:09.911163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:09.911241 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:09.911255 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 15:21:09.911306 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 15:21:09.911369 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"mutex_wait_us":46,"thread_start_us":58,"threads_started":1}
I20260925 15:21:09.911495 14622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:09.906609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41684 (local address 127.3.211.194:46777)
0925 15:21:09.906793 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:09.906798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:09.906810 (+ 12us) server_negotiation.cc:407] Connection header received
0925 15:21:09.907203 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:09.907207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:09.907224 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:09.907537 (+ 313us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:21:09.907606 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:09.908732 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:09.910510 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:09.911034 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:09.911149 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:09.911215 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:09.911362 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 15:21:09.911410 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":96,"threads_started":1}
I20260925 15:21:09.911479 14621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:09.906470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.194:46777 (local address 127.0.0.1:41684)
0925 15:21:09.906630 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:09.906677 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:09.906781 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:09.907359 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:09.907361 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:09.907365 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:21:09.907460 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:09.907463 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:09.908942 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:09.908943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:09.910411 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:09.910413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:09.911149 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:09.911150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:09.911239 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:09.911255 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 15:21:09.911339 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 15:21:09.911388 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":46,"thread_start_us":84,"threads_started":1}
I20260925 15:21:09.911654 14623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:09.906693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:37585 (local address 127.0.0.1:47148)
0925 15:21:09.906813 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:09.906823 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:09.906872 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:09.907418 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:09.907418 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:09.907421 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:21:09.907460 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:09.907461 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:09.908530 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:09.908531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:09.910302 (+ 1771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:09.910305 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:09.911291 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:09.911292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:09.911364 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:09.911367 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:21:09.911413 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 15:21:09.911450 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":56,"thread_start_us":80,"threads_started":1}
I20260925 15:21:09.912071 14625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:09.906977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32792 (local address 127.3.211.195:38649)
0925 15:21:09.907171 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:09.907175 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:09.907184 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:21:09.907220 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:09.907222 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:09.907235 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:09.907315 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:21:09.907563 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:09.908712 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:09.910268 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:09.911052 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:09.911844 (+ 792us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:09.911919 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:09.911944 (+ 25us) server_negotiation.cc:299] Negotiation successful
0925 15:21:09.911991 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":122,"threads_started":1}
I20260925 15:21:09.912519 14626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:09.907038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47148 (local address 127.3.211.193:37585)
0925 15:21:09.907195 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:09.907199 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:09.907209 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:21:09.907247 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:09.907254 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:09.907267 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:09.907365 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:21:09.907526 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:09.908411 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:09.910404 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:09.911139 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:09.911202 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:09.911261 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:09.911513 (+ 252us) server_negotiation.cc:299] Negotiation successful
0925 15:21:09.912459 (+ 946us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":94,"threads_started":1}
I20260925 15:21:09.912405 14410 tablet_service.cc:1467] Processing CreateTablet for tablet a9bfe1c351db4c3ba25f5249759c7074 (DEFAULT_TABLE table=test-table [id=13e5a3152bb347089559b2dde13cf84f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:21:09.912802 14410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9bfe1c351db4c3ba25f5249759c7074. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:21:09.912772 14544 tablet_service.cc:1467] Processing CreateTablet for tablet a9bfe1c351db4c3ba25f5249759c7074 (DEFAULT_TABLE table=test-table [id=13e5a3152bb347089559b2dde13cf84f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:21:09.913071 14544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9bfe1c351db4c3ba25f5249759c7074. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:21:09.916886 14628 tablet_bootstrap.cc:492] T a9bfe1c351db4c3ba25f5249759c7074 P 6d4d3c4f4e1441d78cf4fd6e4e9eafed: Bootstrap starting.
I20260925 15:21:09.917228 14276 tablet_service.cc:1467] Processing CreateTablet for tablet a9bfe1c351db4c3ba25f5249759c7074 (DEFAULT_TABLE table=test-table [id=13e5a3152bb347089559b2dde13cf84f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:21:09.917631 14276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9bfe1c351db4c3ba25f5249759c7074. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:21:09.917867 14628 tablet_bootstrap.cc:654] T a9bfe1c351db4c3ba25f5249759c7074 P 6d4d3c4f4e1441d78cf4fd6e4e9eafed: Neither blocks nor log segments found. Creating new log.
I20260925 15:21:09.918431 14628 log.cc:824] T a9bfe1c351db4c3ba25f5249759c7074 P 6d4d3c4f4e1441d78cf4fd6e4e9eafed: Log is configured to *not* fsync() on all Append() calls
I20260925 15:21:09.918900 14627 tablet_bootstrap.cc:492] T a9bfe1c351db4c3ba25f5249759c7074 P 79ae111fcd28421da92d4bec7c94b17a: Bootstrap starting.
I20260925 15:21:09.919538 14627 tablet_bootstrap.cc:654] T a9bfe1c351db4c3ba25f5249759c7074 P 79ae111fcd28421da92d4bec7c94b17a: Neither blocks nor log segments found. Creating new log.
I20260925 15:21:09.920018 14627 log.cc:824] T a9bfe1c351db4c3ba25f5249759c7074 P 79ae111fcd28421da92d4bec7c94b17a: Log is configured to *not* fsync() on all Append() calls
I20260925 15:21:09.920683 14628 tablet_bootstrap.cc:492] T a9bfe1c351db4c3ba25f5249759c7074 P 6d4d3c4f4e1441d78cf4fd6e4e9eafed: No bootstrap required, opened a new log
I20260925 15:21:09.920787 14628 ts_tablet_manager.cc:1397] T a9bfe1c351db4c3ba25f5249759c7074 P 6d4d3c4f4e1441d78cf4fd6e4e9eafed: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 15:21:09.921001 14627 tablet_bootstrap.cc:492] T a9bfe1c351db4c3ba25f5249759c7074 P 79ae111fcd28421da92d4bec7c94b17a: No bootstrap required, opened a new log
I20260925 15:21:09.921087 14627 ts_tablet_manager.cc:1397] T a9bfe1c351db4c3ba25f5249759c7074 P 79ae111fcd28421da92d4bec7c94b17a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:21:09.921443 14631 tablet_bootstrap.cc:492] T a9bfe1c351db4c3ba25f5249759c7074 P 613fa2f8468045a8a59bf14e27a951d2: Bootstrap starting.
I20260925 15:21:09.922166 14631 tablet_bootstrap.cc:654] T a9bfe1c351db4c3ba25f5249759c7074 P 613fa2f8468045a8a59bf14e27a951d2: Neither blocks nor log segments found. Creating new log.
I20260925 15:21:09.922571 14628 raft_consensus.cc:348] T a9bfe1c351db4c3ba25f5249759c7074 P 6d4d3c4f4e1441d78cf4fd6e4e9eafed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4d3c4f4e1441d78cf4fd6e4e9eafed" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 46777 } } peers { permanent_uuid: "613fa2f8468045a8a59bf14e27a951d2" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 37585 } } peers { permanent_uuid: "79ae111fcd28421da92d4bec7c94b17a" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38649 } }
I20260925 15:21:09.922736 14628 raft_consensus.cc:374] T a9bfe1c351db4c3ba25f5249759c7074 P 6d4d3c4f4e1441d78cf4fd6e4e9eafed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:21:09.922780 14628 raft_consensus.cc:729] T a9bfe1c351db4c3ba25f5249759c7074 P 6d4d3c4f4e1441d78cf4fd6e4e9eafed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d4d3c4f4e1441d78cf4fd6e4e9eafed, State: Initialized, Role: FOLLOWER
I20260925 15:21:09.922791 14631 log.cc:824] T a9bfe1c351db4c3ba25f5249759c7074 P 613fa2f8468045a8a59bf14e27a951d2: Log is configured to *not* fsync() on all Append() calls
I20260925 15:21:09.922950 14628 consensus_queue.cc:260] T a9bfe1c351db4c3ba25f5249759c7074 P 6d4d3c4f4e1441d78cf4fd6e4e9eafed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4d3c4f4e1441d78cf4fd6e4e9eafed" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 46777 } } peers { permanent_uuid: "613fa2f8468045a8a59bf14e27a951d2" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 37585 } } peers { permanent_uuid: "79ae111fcd28421da92d4bec7c94b17a" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38649 } }
I20260925 15:21:09.923272 14628 ts_tablet_manager.cc:1428] T a9bfe1c351db4c3ba25f5249759c7074 P 6d4d3c4f4e1441d78cf4fd6e4e9eafed: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 15:21:09.923408 14476 heartbeater.cc:503] Master 127.3.211.254:39343 was elected leader, sending a full tablet report...
I20260925 15:21:09.923866 14631 tablet_bootstrap.cc:492] T a9bfe1c351db4c3ba25f5249759c7074 P 613fa2f8468045a8a59bf14e27a951d2: No bootstrap required, opened a new log
I20260925 15:21:09.923794 14627 raft_consensus.cc:348] T a9bfe1c351db4c3ba25f5249759c7074 P 79ae111fcd28421da92d4bec7c94b17a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4d3c4f4e1441d78cf4fd6e4e9eafed" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 46777 } } peers { permanent_uuid: "613fa2f8468045a8a59bf14e27a951d2" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 37585 } } peers { permanent_uuid: "79ae111fcd28421da92d4bec7c94b17a" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38649 } }
I20260925 15:21:09.923946 14631 ts_tablet_manager.cc:1397] T a9bfe1c351db4c3ba25f5249759c7074 P 613fa2f8468045a8a59bf14e27a951d2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 15:21:09.923955 14627 raft_consensus.cc:374] T a9bfe1c351db4c3ba25f5249759c7074 P 79ae111fcd28421da92d4bec7c94b17a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:21:09.923991 14627 raft_consensus.cc:729] T a9bfe1c351db4c3ba25f5249759c7074 P 79ae111fcd28421da92d4bec7c94b17a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79ae111fcd28421da92d4bec7c94b17a, State: Initialized, Role: FOLLOWER
I20260925 15:21:09.924126 14627 consensus_queue.cc:260] T a9bfe1c351db4c3ba25f5249759c7074 P 79ae111fcd28421da92d4bec7c94b17a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4d3c4f4e1441d78cf4fd6e4e9eafed" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 46777 } } peers { permanent_uuid: "613fa2f8468045a8a59bf14e27a951d2" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 37585 } } peers { permanent_uuid: "79ae111fcd28421da92d4bec7c94b17a" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38649 } }
I20260925 15:21:09.924453 14627 ts_tablet_manager.cc:1428] T a9bfe1c351db4c3ba25f5249759c7074 P 79ae111fcd28421da92d4bec7c94b17a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:21:09.924490 14610 heartbeater.cc:503] Master 127.3.211.254:39343 was elected leader, sending a full tablet report...
I20260925 15:21:09.926007 14631 raft_consensus.cc:348] T a9bfe1c351db4c3ba25f5249759c7074 P 613fa2f8468045a8a59bf14e27a951d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4d3c4f4e1441d78cf4fd6e4e9eafed" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 46777 } } peers { permanent_uuid: "613fa2f8468045a8a59bf14e27a951d2" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 37585 } } peers { permanent_uuid: "79ae111fcd28421da92d4bec7c94b17a" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38649 } }
I20260925 15:21:09.926146 14631 raft_consensus.cc:374] T a9bfe1c351db4c3ba25f5249759c7074 P 613fa2f8468045a8a59bf14e27a951d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:21:09.926170 14631 raft_consensus.cc:729] T a9bfe1c351db4c3ba25f5249759c7074 P 613fa2f8468045a8a59bf14e27a951d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 613fa2f8468045a8a59bf14e27a951d2, State: Initialized, Role: FOLLOWER
I20260925 15:21:09.926317 14631 consensus_queue.cc:260] T a9bfe1c351db4c3ba25f5249759c7074 P 613fa2f8468045a8a59bf14e27a951d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4d3c4f4e1441d78cf4fd6e4e9eafed" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 46777 } } peers { permanent_uuid: "613fa2f8468045a8a59bf14e27a951d2" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 37585 } } peers { permanent_uuid: "79ae111fcd28421da92d4bec7c94b17a" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38649 } }
I20260925 15:21:09.926641 14631 ts_tablet_manager.cc:1428] T a9bfe1c351db4c3ba25f5249759c7074 P 613fa2f8468045a8a59bf14e27a951d2: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 15:21:09.926653 14342 heartbeater.cc:503] Master 127.3.211.254:39343 was elected leader, sending a full tablet report...
W20260925 15:21:09.960606 14477 tablet.cc:2369] T a9bfe1c351db4c3ba25f5249759c7074 P 6d4d3c4f4e1441d78cf4fd6e4e9eafed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:21:10.002785 14634 raft_consensus.cc:482] T a9bfe1c351db4c3ba25f5249759c7074 P 79ae111fcd28421da92d4bec7c94b17a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:21:10.002954 14634 raft_consensus.cc:504] T a9bfe1c351db4c3ba25f5249759c7074 P 79ae111fcd28421da92d4bec7c94b17a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4d3c4f4e1441d78cf4fd6e4e9eafed" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 46777 } } peers { permanent_uuid: "613fa2f8468045a8a59bf14e27a951d2" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 37585 } } peers { permanent_uuid: "79ae111fcd28421da92d4bec7c94b17a" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38649 } }
I20260925 15:21:10.003376 14634 leader_election.cc:288] T a9bfe1c351db4c3ba25f5249759c7074 P 79ae111fcd28421da92d4bec7c94b17a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d4d3c4f4e1441d78cf4fd6e4e9eafed (127.3.211.194:46777), 613fa2f8468045a8a59bf14e27a951d2 (127.3.211.193:37585)
I20260925 15:21:10.007014 14612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:10.003579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.194:46777 (local address 127.3.211.195:43731)
0925 15:21:10.003633 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:10.003645 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:10.003693 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:10.003916 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:10.003917 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:10.003918 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:21:10.003986 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:10.003988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:10.004790 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:10.004792 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:10.005674 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:10.005676 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:10.006674 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:10.006675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:10.006781 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:10.006787 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 15:21:10.006874 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 15:21:10.006924 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 15:21:10.007051 14622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:10.003720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:43731 (local address 127.3.211.194:46777)
0925 15:21:10.003773 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:10.003775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:10.003787 (+ 12us) server_negotiation.cc:407] Connection header received
0925 15:21:10.003819 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:10.003820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:10.003830 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:10.003908 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:21:10.004088 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:10.004693 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:10.005878 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:10.006545 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:10.006628 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:10.006639 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:10.006926 (+ 287us) server_negotiation.cc:299] Negotiation successful
0925 15:21:10.006994 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 15:21:10.007505 14636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:10.003715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:37585 (local address 127.3.211.195:57509)
0925 15:21:10.003948 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:10.003959 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:10.004019 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:10.004177 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:10.004177 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:10.004179 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:21:10.004263 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:10.004265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:10.005093 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:10.005094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:10.006154 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:10.006160 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:10.007245 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:10.007246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:10.007315 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:10.007319 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:21:10.007372 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 15:21:10.007428 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":146,"threads_started":1}
I20260925 15:21:10.007521 14626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:10.003794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:57509 (local address 127.3.211.193:37585)
0925 15:21:10.003864 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:10.003866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:10.004038 (+ 172us) server_negotiation.cc:407] Connection header received
0925 15:21:10.004096 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:10.004097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:10.004106 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:10.004159 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:21:10.004368 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:10.004981 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:10.006327 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:10.007128 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:10.007207 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:10.007217 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:10.007401 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 15:21:10.007444 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 15:21:10.007462 14430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9bfe1c351db4c3ba25f5249759c7074" candidate_uuid: "79ae111fcd28421da92d4bec7c94b17a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4d3c4f4e1441d78cf4fd6e4e9eafed" is_pre_election: true
I20260925 15:21:10.007658 14430 raft_consensus.cc:2454] T a9bfe1c351db4c3ba25f5249759c7074 P 6d4d3c4f4e1441d78cf4fd6e4e9eafed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79ae111fcd28421da92d4bec7c94b17a in term 0.
I20260925 15:21:10.007776 14296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9bfe1c351db4c3ba25f5249759c7074" candidate_uuid: "79ae111fcd28421da92d4bec7c94b17a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "613fa2f8468045a8a59bf14e27a951d2" is_pre_election: true
I20260925 15:21:10.007939 14296 raft_consensus.cc:2454] T a9bfe1c351db4c3ba25f5249759c7074 P 613fa2f8468045a8a59bf14e27a951d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79ae111fcd28421da92d4bec7c94b17a in term 0.
I20260925 15:21:10.007968 14497 leader_election.cc:302] T a9bfe1c351db4c3ba25f5249759c7074 P 79ae111fcd28421da92d4bec7c94b17a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d4d3c4f4e1441d78cf4fd6e4e9eafed, 79ae111fcd28421da92d4bec7c94b17a; no voters:
I20260925 15:21:10.008208 14634 raft_consensus.cc:2781] T a9bfe1c351db4c3ba25f5249759c7074 P 79ae111fcd28421da92d4bec7c94b17a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:21:10.008276 14634 raft_consensus.cc:482] T a9bfe1c351db4c3ba25f5249759c7074 P 79ae111fcd28421da92d4bec7c94b17a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:21:10.008327 14634 raft_consensus.cc:3037] T a9bfe1c351db4c3ba25f5249759c7074 P 79ae111fcd28421da92d4bec7c94b17a [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:21:10.009092 14634 raft_consensus.cc:504] T a9bfe1c351db4c3ba25f5249759c7074 P 79ae111fcd28421da92d4bec7c94b17a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4d3c4f4e1441d78cf4fd6e4e9eafed" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 46777 } } peers { permanent_uuid: "613fa2f8468045a8a59bf14e27a951d2" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 37585 } } peers { permanent_uuid: "79ae111fcd28421da92d4bec7c94b17a" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38649 } }
I20260925 15:21:10.009244 14634 leader_election.cc:288] T a9bfe1c351db4c3ba25f5249759c7074 P 79ae111fcd28421da92d4bec7c94b17a [CANDIDATE]: Term 1 election: Requested vote from peers 6d4d3c4f4e1441d78cf4fd6e4e9eafed (127.3.211.194:46777), 613fa2f8468045a8a59bf14e27a951d2 (127.3.211.193:37585)
I20260925 15:21:10.009397 14430 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9bfe1c351db4c3ba25f5249759c7074" candidate_uuid: "79ae111fcd28421da92d4bec7c94b17a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d4d3c4f4e1441d78cf4fd6e4e9eafed"
I20260925 15:21:10.009461 14296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9bfe1c351db4c3ba25f5249759c7074" candidate_uuid: "79ae111fcd28421da92d4bec7c94b17a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "613fa2f8468045a8a59bf14e27a951d2"
I20260925 15:21:10.009496 14430 raft_consensus.cc:3037] T a9bfe1c351db4c3ba25f5249759c7074 P 6d4d3c4f4e1441d78cf4fd6e4e9eafed [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:21:10.009542 14296 raft_consensus.cc:3037] T a9bfe1c351db4c3ba25f5249759c7074 P 613fa2f8468045a8a59bf14e27a951d2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:21:10.010423 14296 raft_consensus.cc:2454] T a9bfe1c351db4c3ba25f5249759c7074 P 613fa2f8468045a8a59bf14e27a951d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79ae111fcd28421da92d4bec7c94b17a in term 1.
I20260925 15:21:10.010427 14430 raft_consensus.cc:2454] T a9bfe1c351db4c3ba25f5249759c7074 P 6d4d3c4f4e1441d78cf4fd6e4e9eafed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79ae111fcd28421da92d4bec7c94b17a in term 1.
I20260925 15:21:10.010672 14497 leader_election.cc:302] T a9bfe1c351db4c3ba25f5249759c7074 P 79ae111fcd28421da92d4bec7c94b17a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 613fa2f8468045a8a59bf14e27a951d2, 79ae111fcd28421da92d4bec7c94b17a; no voters:
I20260925 15:21:10.010803 14634 raft_consensus.cc:2781] T a9bfe1c351db4c3ba25f5249759c7074 P 79ae111fcd28421da92d4bec7c94b17a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:21:10.010977 14634 raft_consensus.cc:686] T a9bfe1c351db4c3ba25f5249759c7074 P 79ae111fcd28421da92d4bec7c94b17a [term 1 LEADER]: Becoming Leader. State: Replica: 79ae111fcd28421da92d4bec7c94b17a, State: Running, Role: LEADER
I20260925 15:21:10.011083 14634 consensus_queue.cc:237] T a9bfe1c351db4c3ba25f5249759c7074 P 79ae111fcd28421da92d4bec7c94b17a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4d3c4f4e1441d78cf4fd6e4e9eafed" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 46777 } } peers { permanent_uuid: "613fa2f8468045a8a59bf14e27a951d2" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 37585 } } peers { permanent_uuid: "79ae111fcd28421da92d4bec7c94b17a" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38649 } }
I20260925 15:21:10.012118 14150 catalog_manager.cc:5495] T a9bfe1c351db4c3ba25f5249759c7074 P 79ae111fcd28421da92d4bec7c94b17a reported cstate change: term changed from 0 to 1, leader changed from <none> to 79ae111fcd28421da92d4bec7c94b17a (127.3.211.195). New cstate: current_term: 1 leader_uuid: "79ae111fcd28421da92d4bec7c94b17a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d4d3c4f4e1441d78cf4fd6e4e9eafed" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 46777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "613fa2f8468045a8a59bf14e27a951d2" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 37585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79ae111fcd28421da92d4bec7c94b17a" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 38649 } health_report { overall_health: HEALTHY } } }
I20260925 15:21:10.028578 14625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:10.026184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32800 (local address 127.3.211.195:38649)
0925 15:21:10.026245 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:10.026247 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:10.026256 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:21:10.026306 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:10.026307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:10.026325 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:10.026381 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:21:10.026543 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:10.027031 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:10.027761 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:10.027947 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:10.028009 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:10.028143 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:21:10.028248 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:21:10.028311 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:10.028410 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 15:21:10.028439 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 15:21:10.030421 14296 raft_consensus.cc:1261] T a9bfe1c351db4c3ba25f5249759c7074 P 613fa2f8468045a8a59bf14e27a951d2 [term 1 FOLLOWER]: Refusing update from remote peer 79ae111fcd28421da92d4bec7c94b17a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:21:10.030447 14430 raft_consensus.cc:1261] T a9bfe1c351db4c3ba25f5249759c7074 P 6d4d3c4f4e1441d78cf4fd6e4e9eafed [term 1 FOLLOWER]: Refusing update from remote peer 79ae111fcd28421da92d4bec7c94b17a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:21:10.030823 14634 consensus_queue.cc:1035] T a9bfe1c351db4c3ba25f5249759c7074 P 79ae111fcd28421da92d4bec7c94b17a [LEADER]: Connected to new peer: Peer: permanent_uuid: "613fa2f8468045a8a59bf14e27a951d2" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:21:10.030997 14637 consensus_queue.cc:1035] T a9bfe1c351db4c3ba25f5249759c7074 P 79ae111fcd28421da92d4bec7c94b17a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d4d3c4f4e1441d78cf4fd6e4e9eafed" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 46777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:21:10.033582 14640 mvcc.cc:205] Tried to move back new op lower bound from 7333272248441802752 to 7333272248366321664. Current Snapshot: MvccSnapshot[applied={T|T < 7333272248441802752}]
I20260925 15:21:10.033967 14641 mvcc.cc:205] Tried to move back new op lower bound from 7333272248441802752 to 7333272248366321664. Current Snapshot: MvccSnapshot[applied={T|T < 7333272248441802752}]
I20260925 15:21:10.034818 14642 mvcc.cc:205] Tried to move back new op lower bound from 7333272248441802752 to 7333272248366321664. Current Snapshot: MvccSnapshot[applied={T|T < 7333272248441802752 or (T in {7333272248441802752})}]
I20260925 15:21:10.037091 14151 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:21:10.037210 14151 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:21:10.038095 14151 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6f13fd2c0b8542cfabcf036f7fbf272b: Sending DeleteTablet for 3 replicas of tablet a9bfe1c351db4c3ba25f5249759c7074
I20260925 15:21:10.038527 14544 tablet_service.cc:1514] Processing DeleteTablet for tablet a9bfe1c351db4c3ba25f5249759c7074 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:21:10 UTC) from {username='kudu', principal='kudu/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:32792
I20260925 15:21:10.038546 14410 tablet_service.cc:1514] Processing DeleteTablet for tablet a9bfe1c351db4c3ba25f5249759c7074 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:21:10 UTC) from {username='kudu', principal='kudu/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:41684
I20260925 15:21:10.038789 14276 tablet_service.cc:1514] Processing DeleteTablet for tablet a9bfe1c351db4c3ba25f5249759c7074 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:21:10 UTC) from {username='kudu', principal='kudu/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:47148
I20260925 15:21:10.038820 14650 tablet_replica.cc:330] T a9bfe1c351db4c3ba25f5249759c7074 P 6d4d3c4f4e1441d78cf4fd6e4e9eafed: stopping tablet replica
I20260925 15:21:10.038808 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 14213
I20260925 15:21:10.038930 14650 raft_consensus.cc:2229] T a9bfe1c351db4c3ba25f5249759c7074 P 6d4d3c4f4e1441d78cf4fd6e4e9eafed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:21:10.039040 14650 raft_consensus.cc:2258] T a9bfe1c351db4c3ba25f5249759c7074 P 6d4d3c4f4e1441d78cf4fd6e4e9eafed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:21:10.039707 14651 tablet_replica.cc:330] T a9bfe1c351db4c3ba25f5249759c7074 P 79ae111fcd28421da92d4bec7c94b17a: stopping tablet replica
I20260925 15:21:10.039803 14651 raft_consensus.cc:2229] T a9bfe1c351db4c3ba25f5249759c7074 P 79ae111fcd28421da92d4bec7c94b17a [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:21:10.039928 14651 raft_consensus.cc:2258] T a9bfe1c351db4c3ba25f5249759c7074 P 79ae111fcd28421da92d4bec7c94b17a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:21:10.040477 14651 ts_tablet_manager.cc:1905] T a9bfe1c351db4c3ba25f5249759c7074 P 79ae111fcd28421da92d4bec7c94b17a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:21:10.040491 14650 ts_tablet_manager.cc:1905] T a9bfe1c351db4c3ba25f5249759c7074 P 6d4d3c4f4e1441d78cf4fd6e4e9eafed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:21:10.042357 14651 ts_tablet_manager.cc:1918] T a9bfe1c351db4c3ba25f5249759c7074 P 79ae111fcd28421da92d4bec7c94b17a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:21:10.042433 14651 log.cc:1196] T a9bfe1c351db4c3ba25f5249759c7074 P 79ae111fcd28421da92d4bec7c94b17a: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/ts-2/wal/wals/a9bfe1c351db4c3ba25f5249759c7074
I20260925 15:21:10.042529 14650 ts_tablet_manager.cc:1918] T a9bfe1c351db4c3ba25f5249759c7074 P 6d4d3c4f4e1441d78cf4fd6e4e9eafed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:21:10.042601 14650 log.cc:1196] T a9bfe1c351db4c3ba25f5249759c7074 P 6d4d3c4f4e1441d78cf4fd6e4e9eafed: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790349610595785-3919-0/minicluster-data/ts-1/wal/wals/a9bfe1c351db4c3ba25f5249759c7074
I20260925 15:21:10.042721 14651 ts_tablet_manager.cc:1939] T a9bfe1c351db4c3ba25f5249759c7074 P 79ae111fcd28421da92d4bec7c94b17a: Deleting consensus metadata
I20260925 15:21:10.042917 14650 ts_tablet_manager.cc:1939] T a9bfe1c351db4c3ba25f5249759c7074 P 6d4d3c4f4e1441d78cf4fd6e4e9eafed: Deleting consensus metadata
I20260925 15:21:10.043200 14135 catalog_manager.cc:4841] TS 79ae111fcd28421da92d4bec7c94b17a (127.3.211.195:38649): tablet a9bfe1c351db4c3ba25f5249759c7074 (table test-table [id=13e5a3152bb347089559b2dde13cf84f]) successfully deleted
I20260925 15:21:10.043401 14135 catalog_manager.cc:4841] TS 6d4d3c4f4e1441d78cf4fd6e4e9eafed (127.3.211.194:46777): tablet a9bfe1c351db4c3ba25f5249759c7074 (table test-table [id=13e5a3152bb347089559b2dde13cf84f]) successfully deleted
W20260925 15:21:10.045979 14135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.211.193:37585 (error 108)
I20260925 15:21:10.046603 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 14347
I20260925 15:21:10.046723 14623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:10.046450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:37585 (local address 127.0.0.1:47156)
0925 15:21:10.046634 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:10.046685 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.193:37585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
W20260925 15:21:10.048926 14135 catalog_manager.cc:4584] TS 613fa2f8468045a8a59bf14e27a951d2 (127.3.211.193:37585): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a9bfe1c351db4c3ba25f5249759c7074: Network error: Client connection negotiation failed: client connection to 127.3.211.193:37585: connect: Connection refused (error 111)
I20260925 15:21:10.052989 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 14481
I20260925 15:21:10.058110 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 14119
2026-09-25T15:21:10Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2075 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:21:10 dist-test-slave-4bk3 krb5kdc[14655](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:21:10 dist-test-slave-4bk3 krb5kdc[14655](info): set up 2 sockets
Sep 25 15:21:10 dist-test-slave-4bk3 krb5kdc[14655](info): commencing operation
krb5kdc: starting...
W20260925 15:21:11.109350 3919 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Sep 25 15:21:11 dist-test-slave-4bk3 krb5kdc[14655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:21:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:21:11Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.211.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.211.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.211.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/krb5kdc/rangeradmin_127.3.211.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.211.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/krb5kdc/rangeradmin_127.3.211.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.211.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.211.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.211.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/krb5kdc/rangerlookup_127.3.211.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.211.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/krb5kdc/rangerlookup_127.3.211.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.211.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.211.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.211.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/krb5kdc/HTTP_127.3.211.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.211.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/krb5kdc/HTTP_127.3.211.212@KRBTEST.COM.keytab.
I20260925 15:21:11.283450 3919 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskAzNr2f/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/postgres -l logfile start
2026-09-25 15:21:14.081 UTC [14685] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 15:21:14.081 UTC [14685] LOG: listening on IPv4 address "127.3.211.212", port 40211
2026-09-25 15:21:14.085 UTC [14685] LOG: listening on Unix socket "/tmp/.s.PGSQL.40211"
2026-09-25 15:21:14.121 UTC [14688] LOG: database system was shut down at 2026-09-25 15:21:12 UTC
2026-09-25 15:21:14.126 UTC [14685] LOG: database system is ready to accept connections
I20260925 15:21:15.085247 3919 mini_postgres.cc:96] Postgres bound to 40211
2026-09-25 15:21:15.091 UTC [14696] FATAL: database "slave" does not exist
127.3.211.212:40211 - accepting connections
I20260925 15:21:15.092626 3919 mini_ranger.cc:161] Starting Ranger...
I20260925 15:21:15.108814 3919 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 15:21:15.662247 3919 mini_ranger.cc:87] Created ranger Postgres database
I20260925 15:21:15.662341 3919 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ranger-admin
2026-09-25 15:21:15,959 [I] DB FLAVOR :POSTGRES
2026-09-25 15:21:15,960 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 15:21:15,960 [I] Checking connection..
2026-09-25 15:21:15,960 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNr2f/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNr2f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.211.212:40211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 15:21:16,233 [I] Checking connection passed.
2026-09-25 15:21:16,233 [I] --------- Verifying version history table ---------
2026-09-25 15:21:16,234 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNr2f/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNr2f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.211.212:40211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 15:21:16,484 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 15:21:16,484 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNr2f/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNr2f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.211.212:40211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 15:21:16,728 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 15:21:16,728 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 15:21:16,728 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNr2f/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNr2f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.211.212:40211/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAzNr2f/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 15:21:16.990 UTC [14760] WARNING: there is no transaction in progress
2026-09-25 15:21:16,997 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 15:21:16,997 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNr2f/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNr2f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.211.212:40211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 15:21:17,258 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 15:21:17,259 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 15:21:17,259 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNr2f/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNr2f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.211.212:40211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 15:21:17,516 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNr2f/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNr2f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.211.212:40211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 15:21:17,794 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNr2f/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNr2f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.211.212:40211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-4bk3.c.gcp-upstream.internal','N') ;"
2026-09-25 15:21:18,044 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNr2f/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNr2f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.211.212:40211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 15:21:18,304 [I] Table x_portal_user does not exist in database ranger
2026-09-25 15:21:18,304 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNr2f/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNr2f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.211.212:40211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 15:21:18,573 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 15:21:18,574 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNr2f/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNr2f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.211.212:40211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 15:21:18,827 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNr2f/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNr2f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.211.212:40211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 15:21:19,072 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 15:21:19,073 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNr2f/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNr2f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.211.212:40211/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAzNr2f/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 15:21:20.143 UTC [14868] WARNING: there is no transaction in progress
2026-09-25 15:21:20.161 UTC [14868] WARNING: there is no transaction in progress
2026-09-25 15:21:20.184 UTC [14868] WARNING: there is no transaction in progress
2026-09-25 15:21:20.203 UTC [14868] WARNING: there is no transaction in progress
2026-09-25 15:21:20.220 UTC [14868] WARNING: there is no transaction in progress
2026-09-25 15:21:20.240 UTC [14868] WARNING: there is no transaction in progress
2026-09-25 15:21:20.305 UTC [14868] WARNING: there is no transaction in progress
2026-09-25 15:21:20.312 UTC [14868] WARNING: there is no transaction in progress
2026-09-25 15:21:20.328 UTC [14868] WARNING: there is no transaction in progress
2026-09-25 15:21:20.340 UTC [14868] WARNING: there is no transaction in progress
2026-09-25 15:21:20.352 UTC [14868] WARNING: there is no transaction in progress
2026-09-25 15:21:20.368 UTC [14868] WARNING: there is no transaction in progress
2026-09-25 15:21:20.381 UTC [14868] WARNING: there is no transaction in progress
2026-09-25 15:21:20.391 UTC [14868] WARNING: there is no transaction in progress
2026-09-25 15:21:20.401 UTC [14868] WARNING: there is no transaction in progress
2026-09-25 15:21:20.410 UTC [14868] WARNING: there is no transaction in progress
2026-09-25 15:21:21,243 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 15:21:21,243 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNr2f/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNr2f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.211.212:40211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 15:21:21,490 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 15:21:21,490 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNr2f/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNr2f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.211.212:40211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 15:21:21,736 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 15:21:21,736 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNr2f/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNr2f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.211.212:40211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 15:21:21,993 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 15:21:21,993 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNr2f/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNr2f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.211.212:40211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 15:21:22,257 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNr2f/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNr2f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.211.212:40211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 15:21:22,511 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNr2f/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNr2f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.211.212:40211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-4bk3.c.gcp-upstream.internal';"
2026-09-25 15:21:22,751 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 15:21:22,752 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNr2f/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNr2f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.211.212:40211/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 15:21:23,022 [I] DB_PATCHES have already been applied
I20260925 15:21:23.027786 3919 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ranger-admin:/tmp/dist-test-taskAzNr2f/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskAzNr2f/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskAzNr2f/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNr2f/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 15:21:23.027851 3919 mini_ranger.cc:193] Using host: 127.3.211.212
I20260925 15:21:23.030949 3919 mini_ranger.cc:227] Ranger admin URL: http://127.3.211.212:34995
Sep 25, 2026 3:21:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 3:21:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 3:21:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 3:21:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 3:21:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 3:21:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 3:21:24 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 15:21:24.151947 3919 mini_ranger.cc:160] Time spent starting Ranger: real 9.059s user 0.002s sys 0.007s
Sep 25, 2026 3:21:45 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 3:21:45 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 25, 2026 3:21:45 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 3:21:45 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 3:21:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 3:21:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 3:21:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 3:21:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 3:21:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 3:21:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 3:21:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 3:21:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 3:21:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 3:21:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 3:21:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 3:21:45 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 3:21:45 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 3:21:46 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 15:21:46.920846 3919 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.211.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.211.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.211.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/krb5kdc/rangerkms_127.3.211.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.211.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/krb5kdc/rangerkms_127.3.211.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.211.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.211.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.211.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/krb5kdc/HTTP_127.3.211.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.211.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/krb5kdc/HTTP_127.3.211.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 15:21:46 dist-test-slave-4bk3 krb5kdc[14655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349706, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 15:21:47.007463 3919 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 15:21:47.018767 3919 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 15:21:48.041819 3919 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 15:21:48.041909 3919 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ranger-kms
I20260925 15:21:48.041929 3919 mini_ranger_kms.cc:227] Using postgres at 127.3.211.212:40211
2026-09-25 15:21:48,233 [I] DB FLAVOR :POSTGRES
2026-09-25 15:21:48,234 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 15:21:48,234 [I] Checking connection
2026-09-25 15:21:48,234 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNr2f/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.211.212:40211/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 15:21:48,504 [I] connection success
2026-09-25 15:21:48,504 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 15:21:48,504 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 15:21:48,504 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNr2f/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.211.212:40211/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 15:21:48,771 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 15:21:48,771 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 15:21:48,771 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 15:21:48,771 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNr2f/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.211.212:40211/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 15:21:49,055 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 15:21:49.608964 3919 mini_ranger_kms.cc:310] Created kms service
I20260925 15:21:49.865762 3919 mini_ranger_kms.cc:326] Created kudu user
I20260925 15:21:49.917071 3919 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 15:21:49 dist-test-slave-4bk3 krb5kdc[14655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349706, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.211.212@KRBTEST.COM
I20260925 15:21:50.367709 3919 mini_ranger_kms.cc:384] Added ranger policy
I20260925 15:21:50.367836 3919 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ranger-kms:/tmp/dist-test-taskAzNr2f/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNr2f/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskAzNr2f/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 15:21:50.367861 3919 mini_ranger_kms.cc:240] Using host: 127.3.211.212
I20260925 15:21:50.371671 3919 mini_ranger_kms.cc:276] Ranger KMS PID: 15049
I20260925 15:21:50.371824 3919 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.211.212:41087
Sep 25, 2026 3:21:51 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 3:21:51 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 3:21:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 3:21:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 3:21:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 3:21:51 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 15:21:52.625260 3919 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.618s user 0.003s sys 0.166s
I20260925 15:21:52.625368 3919 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 15:21:52.625440 3919 mini_ranger_kms.cc:401] 127.3.211.212:41087/kms/v1/keys
Sep 25 15:21:52 dist-test-slave-4bk3 krb5kdc[14655](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790349712, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.211.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 15:21:53 dist-test-slave-4bk3 krb5kdc[14655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349712, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.211.212@KRBTEST.COM for HTTP/127.3.211.212@KRBTEST.COM
Sep 25 15:21:53 dist-test-slave-4bk3 krb5kdc[14655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349712, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.211.212@KRBTEST.COM for HTTP/127.3.211.212@KRBTEST.COM
Sep 25 15:21:55 dist-test-slave-4bk3 krb5kdc[14655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.211.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
I20260925 15:21:55.884420 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:33937
--webserver_interface=127.3.211.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:33543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.211.254:33937
--ranger_config_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.211.212:41087/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:21:55.972321 15099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:21:55.972687 15099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:21:55.972728 15099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:21:55.974058 15099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:21:55.974152 15099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:21:55.974176 15099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:21:55.974200 15099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:21:55.974233 15099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:21:55.976244 15099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:33543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.211.212:41087/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.211.254:33937
--ranger_config_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:33937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.211.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:21:55.976606 15099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:21:55.976876 15099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:21:55.982093 15105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:21:55.982098 15104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:21:55.982093 15107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:21:55.982527 15099 server_base.cc:1034] running on GCE node
I20260925 15:21:55.982797 15099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:21:55.983146 15099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:21:55.984354 15099 hybrid_clock.cc:648] HybridClock initialized: now 1790349715984301 us; error 43 us; skew 500 ppm
Sep 25 15:21:55 dist-test-slave-4bk3 krb5kdc[14655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349715, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:21:55.990756 15099 init.cc:377] Logged in from keytab as kudu/127.3.211.254@KRBTEST.COM (short username kudu)
I20260925 15:21:55.991335 15099 webserver.cc:460] Webserver started at http://127.3.211.254:34897/ using document root <none> and password file <none>
I20260925 15:21:55.991598 15099 fs_manager.cc:359] Metadata directory not provided
I20260925 15:21:55.991662 15099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:21:55.991827 15099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:21:55 dist-test-slave-4bk3 krb5kdc[14655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349715, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.254@KRBTEST.COM for HTTP/127.3.211.212@KRBTEST.COM
I20260925 15:21:56.049106 15099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/master-0/data/instance:
uuid: "8452a69e9a5b4ce5b33e0ade56b99a18"
format_stamp: "Formatted at 2026-09-25 15:21:56 on dist-test-slave-4bk3"
server_key: "6794e3e966c76fd8dc09c039df8dd9df"
server_key_iv: "a98347b235bf0d2dad024428927d8617"
server_key_version: "kuduclusterkey@0"
I20260925 15:21:56.049566 15099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/master-0/wal/instance:
uuid: "8452a69e9a5b4ce5b33e0ade56b99a18"
format_stamp: "Formatted at 2026-09-25 15:21:56 on dist-test-slave-4bk3"
server_key: "6794e3e966c76fd8dc09c039df8dd9df"
server_key_iv: "a98347b235bf0d2dad024428927d8617"
server_key_version: "kuduclusterkey@0"
I20260925 15:21:56.051446 15099 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:21:56.064724 15114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:56.064910 15099 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 15:21:56.065030 15099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/master-0/wal
uuid: "8452a69e9a5b4ce5b33e0ade56b99a18"
format_stamp: "Formatted at 2026-09-25 15:21:56 on dist-test-slave-4bk3"
server_key: "6794e3e966c76fd8dc09c039df8dd9df"
server_key_iv: "a98347b235bf0d2dad024428927d8617"
server_key_version: "kuduclusterkey@0"
I20260925 15:21:56.065165 15099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:21:56.086972 15099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:21:56.089260 15099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:21:56.089444 15099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:21:56.094177 15099 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.254:33937
I20260925 15:21:56.094200 15166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.254:33937 every 8 connection(s)
I20260925 15:21:56.094580 15099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
I20260925 15:21:56.095422 15167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:21:56.099159 15167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8452a69e9a5b4ce5b33e0ade56b99a18: Bootstrap starting.
I20260925 15:21:56.099910 15167 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8452a69e9a5b4ce5b33e0ade56b99a18: Neither blocks nor log segments found. Creating new log.
I20260925 15:21:56.100234 15167 log.cc:824] T 00000000000000000000000000000000 P 8452a69e9a5b4ce5b33e0ade56b99a18: Log is configured to *not* fsync() on all Append() calls
I20260925 15:21:56.101089 15167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8452a69e9a5b4ce5b33e0ade56b99a18: No bootstrap required, opened a new log
I20260925 15:21:56.101692 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 15099
I20260925 15:21:56.101794 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/master-0/wal/instance
I20260925 15:21:56.101971 3919 external_mini_cluster.cc:1428] Setting key 4dbec9c34ced45f2f623ea13f5a7f3f5
I20260925 15:21:56.103993 15167 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8452a69e9a5b4ce5b33e0ade56b99a18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8452a69e9a5b4ce5b33e0ade56b99a18" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 33937 } }
I20260925 15:21:56.104183 15167 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8452a69e9a5b4ce5b33e0ade56b99a18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:21:56.104218 15167 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8452a69e9a5b4ce5b33e0ade56b99a18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8452a69e9a5b4ce5b33e0ade56b99a18, State: Initialized, Role: FOLLOWER
I20260925 15:21:56.104432 15167 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8452a69e9a5b4ce5b33e0ade56b99a18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8452a69e9a5b4ce5b33e0ade56b99a18" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 33937 } }
I20260925 15:21:56.104517 15167 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8452a69e9a5b4ce5b33e0ade56b99a18 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:21:56.104673 15167 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8452a69e9a5b4ce5b33e0ade56b99a18 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:21:56.104784 15167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8452a69e9a5b4ce5b33e0ade56b99a18 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:21:56.105520 15167 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8452a69e9a5b4ce5b33e0ade56b99a18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8452a69e9a5b4ce5b33e0ade56b99a18" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 33937 } }
I20260925 15:21:56.105628 15167 leader_election.cc:302] T 00000000000000000000000000000000 P 8452a69e9a5b4ce5b33e0ade56b99a18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8452a69e9a5b4ce5b33e0ade56b99a18; no voters:
I20260925 15:21:56.105827 15167 leader_election.cc:288] T 00000000000000000000000000000000 P 8452a69e9a5b4ce5b33e0ade56b99a18 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:21:56.105890 15172 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8452a69e9a5b4ce5b33e0ade56b99a18 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:21:56.106134 15172 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8452a69e9a5b4ce5b33e0ade56b99a18 [term 1 LEADER]: Becoming Leader. State: Replica: 8452a69e9a5b4ce5b33e0ade56b99a18, State: Running, Role: LEADER
I20260925 15:21:56.106190 15167 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8452a69e9a5b4ce5b33e0ade56b99a18 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:21:56.106305 15172 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8452a69e9a5b4ce5b33e0ade56b99a18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8452a69e9a5b4ce5b33e0ade56b99a18" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 33937 } }
I20260925 15:21:56.106698 15172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8452a69e9a5b4ce5b33e0ade56b99a18 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8452a69e9a5b4ce5b33e0ade56b99a18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8452a69e9a5b4ce5b33e0ade56b99a18" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 33937 } } }
I20260925 15:21:56.106842 15172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8452a69e9a5b4ce5b33e0ade56b99a18 [sys.catalog]: This master's current role is: LEADER
I20260925 15:21:56.107025 15172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8452a69e9a5b4ce5b33e0ade56b99a18 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8452a69e9a5b4ce5b33e0ade56b99a18. Latest consensus state: current_term: 1 leader_uuid: "8452a69e9a5b4ce5b33e0ade56b99a18" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8452a69e9a5b4ce5b33e0ade56b99a18" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 33937 } } }
I20260925 15:21:56.107134 15172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8452a69e9a5b4ce5b33e0ade56b99a18 [sys.catalog]: This master's current role is: LEADER
I20260925 15:21:56.107301 15181 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:21:56.107872 15181 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:21:56.108132 15167 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Sep 25 15:21:56 dist-test-slave-4bk3 krb5kdc[14655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:21:56.113180 15189 mvcc.cc:205] Tried to move back new op lower bound from 7333272437190918144 to 7333272437171884032. Current Snapshot: MvccSnapshot[applied={T|T < 7333272437190918144}]
I20260925 15:21:56.114359 15181 catalog_manager.cc:1347] Generated new cluster ID: d2ed12e8c24e4c0294c0530af8228402
I20260925 15:21:56.114446 15181 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:21:56.117400 15171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:56.102544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55966 (local address 127.3.211.254:33937)
0925 15:21:56.102742 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:56.102753 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:56.102787 (+ 34us) server_negotiation.cc:407] Connection header received
0925 15:21:56.104002 (+ 1215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:56.104011 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:56.104107 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:56.104303 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:56.105232 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:56.106587 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:56.107737 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:56.108370 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:56.108473 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:56.112039 (+ 3566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:21:56.112090 (+ 51us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:21:56.112091 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:21:56.112184 (+ 93us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:21:56.115215 (+ 3031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:56.115646 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:56.115649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:56.115650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:56.115763 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:56.115950 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:56.115951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:56.115951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:56.116315 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:21:56.116501 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:56.116968 (+ 467us) server_negotiation.cc:299] Negotiation successful
0925 15:21:56.117152 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":70,"threads_started":1}
I20260925 15:21:56.126768 15181 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:21:56.127590 15181 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:21:56.132930 15181 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8452a69e9a5b4ce5b33e0ade56b99a18: Generated new TSK 0
I20260925 15:21:56.133222 15181 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.211.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab.
Entry for principal kudu/127.3.211.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab.
I20260925 15:21:56.972297 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.211.212:41087/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.193:0
--local_ip_for_outbound_sockets=127.3.211.193
--webserver_interface=127.3.211.193
--webserver_port=0
--tserver_master_addrs=127.3.211.254:33937
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab
--principal=kudu/127.3.211.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:33543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:21:57.100236 15212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:21:57.100600 15212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:21:57.100646 15212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:21:57.102042 15212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:21:57.102128 15212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:21:57.102178 15212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.193
I20260925 15:21:57.103870 15212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:33543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.211.212:41087/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab
--principal=kudu/127.3.211.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.211.193
--webserver_port=0
--tserver_master_addrs=127.3.211.254:33937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.193
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:21:57.104169 15212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:21:57.104449 15212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:21:57.110361 15218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:21:57.110541 15212 server_base.cc:1034] running on GCE node
W20260925 15:21:57.110368 15217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:21:57.110723 15220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:21:57.111192 15212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:21:57.111500 15212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:21:57.112754 15212 hybrid_clock.cc:648] HybridClock initialized: now 1790349717112719 us; error 39 us; skew 500 ppm
Sep 25 15:21:57 dist-test-slave-4bk3 krb5kdc[14655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349717, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:21:57.116145 15212 init.cc:377] Logged in from keytab as kudu/127.3.211.193@KRBTEST.COM (short username kudu)
I20260925 15:21:57.116792 15212 webserver.cc:460] Webserver started at http://127.3.211.193:42227/ using document root <none> and password file <none>
I20260925 15:21:57.117146 15212 fs_manager.cc:359] Metadata directory not provided
I20260925 15:21:57.117261 15212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:21:57.117434 15212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:21:57 dist-test-slave-4bk3 krb5kdc[14655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349717, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.193@KRBTEST.COM for HTTP/127.3.211.212@KRBTEST.COM
I20260925 15:21:57.129311 15212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ts-0/data/instance:
uuid: "69d90e8a3597467cb07b5efd7f5a322a"
format_stamp: "Formatted at 2026-09-25 15:21:57 on dist-test-slave-4bk3"
server_key: "e37455d2b6f4fab0fcac5bcec8a63807"
server_key_iv: "564549bf9fac488102f32d621ce3c7f8"
server_key_version: "kuduclusterkey@0"
I20260925 15:21:57.129843 15212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ts-0/wal/instance:
uuid: "69d90e8a3597467cb07b5efd7f5a322a"
format_stamp: "Formatted at 2026-09-25 15:21:57 on dist-test-slave-4bk3"
server_key: "e37455d2b6f4fab0fcac5bcec8a63807"
server_key_iv: "564549bf9fac488102f32d621ce3c7f8"
server_key_version: "kuduclusterkey@0"
I20260925 15:21:57.131855 15212 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:21:57.147257 15227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:57.147507 15212 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:21:57.147624 15212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ts-0/wal
uuid: "69d90e8a3597467cb07b5efd7f5a322a"
format_stamp: "Formatted at 2026-09-25 15:21:57 on dist-test-slave-4bk3"
server_key: "e37455d2b6f4fab0fcac5bcec8a63807"
server_key_iv: "564549bf9fac488102f32d621ce3c7f8"
server_key_version: "kuduclusterkey@0"
I20260925 15:21:57.147735 15212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:21:57.182796 15212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:21:57.185501 15212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:21:57.185760 15212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:21:57.186058 15212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:21:57.186486 15212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:21:57.186585 15212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:57.186681 15212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:21:57.186748 15212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:57.195454 15212 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.193:34243
I20260925 15:21:57.196017 15212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:21:57.198318 15342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.193:34243 every 8 connection(s)
I20260925 15:21:57.202579 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 15212
I20260925 15:21:57.202695 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ts-0/wal/instance
I20260925 15:21:57.202886 3919 external_mini_cluster.cc:1428] Setting key c95e7ff89cded09ad68671e4e28c122d
Sep 25 15:21:57 dist-test-slave-4bk3 krb5kdc[14655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349717, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.193@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:21:57.218516 15345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:57.199020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:33937 (local address 127.3.211.193:35771)
0925 15:21:57.199381 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:57.199427 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:57.200051 (+ 624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:57.200507 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:57.200513 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:57.200813 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:21:57.201106 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:57.201112 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:57.205444 (+ 4332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:57.205446 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:57.206991 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:57.206996 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:57.211808 (+ 4812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:57.211810 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:57.212004 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:57.212759 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:21:57.212771 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:21:57.214557 (+ 1786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:21:57.216786 (+ 2229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:21:57.216795 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:21:57.216798 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:21:57.217130 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:21:57.217371 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:21:57.217372 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:21:57.217372 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:21:57.217506 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:21:57.217791 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:21:57.217797 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:21:57.217971 (+ 174us) client_negotiation.cc:769] Sending connection context
0925 15:21:57.218077 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 15:21:57.218260 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":94,"threads_started":1}
I20260925 15:21:57.219146 15346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:57.199378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:35771 (local address 127.3.211.254:33937)
0925 15:21:57.200249 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:57.200251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:57.200259 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:21:57.200285 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:57.200286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:57.200299 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:57.200402 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:57.203382 (+ 2980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:57.204276 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:57.211217 (+ 6941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:57.211570 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:57.211700 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:57.214710 (+ 3010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:21:57.214712 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:21:57.214713 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:21:57.214723 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:21:57.216660 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:57.217238 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:57.217239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:57.217240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:57.217308 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:57.217569 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:57.217570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:57.217570 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:57.217710 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:21:57.217759 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:57.218927 (+ 1168us) server_negotiation.cc:299] Negotiation successful
0925 15:21:57.219024 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":846,"thread_start_us":57,"threads_started":1}
I20260925 15:21:57.219937 15343 heartbeater.cc:347] Connected to a master server at 127.3.211.254:33937
I20260925 15:21:57.220036 15343 heartbeater.cc:464] Registering TS with master...
I20260925 15:21:57.220245 15343 heartbeater.cc:511] Master 127.3.211.254:33937 requested a full tablet report, sending...
I20260925 15:21:57.220883 15131 ts_manager.cc:194] Registered new tserver with Master: 69d90e8a3597467cb07b5efd7f5a322a (127.3.211.193:34243)
I20260925 15:21:57.221781 15131 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.193@KRBTEST.COM'} at 127.3.211.193:35771
WARNING: no policy specified for kudu/127.3.211.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab.
Entry for principal kudu/127.3.211.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab.
I20260925 15:21:57.255563 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.211.212:41087/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.194:0
--local_ip_for_outbound_sockets=127.3.211.194
--webserver_interface=127.3.211.194
--webserver_port=0
--tserver_master_addrs=127.3.211.254:33937
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab
--principal=kudu/127.3.211.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:33543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:21:57.394886 15349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:21:57.395128 15349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:21:57.395176 15349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:21:57.397210 15349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:21:57.397344 15349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:21:57.397406 15349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.194
I20260925 15:21:57.399168 15349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:33543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.211.212:41087/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab
--principal=kudu/127.3.211.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.211.194
--webserver_port=0
--tserver_master_addrs=127.3.211.254:33937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.194
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:21:57.399591 15349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:21:57.399951 15349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:21:57.406700 15359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:21:57.406836 15357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:21:57.406704 15356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:21:57.407999 15349 server_base.cc:1034] running on GCE node
I20260925 15:21:57.408241 15349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:21:57.408545 15349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
Sep 25 15:21:57 dist-test-slave-4bk3 krb5kdc[14655](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790349717, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:21:57.409966 15349 hybrid_clock.cc:648] HybridClock initialized: now 1790349717409613 us; error 265 us; skew 500 ppm
Sep 25 15:21:57 dist-test-slave-4bk3 krb5kdc[14655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349717, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:21:57.413805 15349 init.cc:377] Logged in from keytab as kudu/127.3.211.194@KRBTEST.COM (short username kudu)
I20260925 15:21:57.414502 15349 webserver.cc:460] Webserver started at http://127.3.211.194:32969/ using document root <none> and password file <none>
I20260925 15:21:57.414825 15349 fs_manager.cc:359] Metadata directory not provided
I20260925 15:21:57.414909 15349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:21:57.415094 15349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:21:57 dist-test-slave-4bk3 krb5kdc[14655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349717, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.194@KRBTEST.COM for HTTP/127.3.211.212@KRBTEST.COM
I20260925 15:21:57.435540 15349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ts-1/data/instance:
uuid: "097e0db38a304bdcb2e0b46b421e65be"
format_stamp: "Formatted at 2026-09-25 15:21:57 on dist-test-slave-4bk3"
server_key: "760f70be026b84aeec490e7b7bd0c258"
server_key_iv: "58985c4becfb05a68d3e2a8febcd4e93"
server_key_version: "kuduclusterkey@0"
I20260925 15:21:57.436318 15349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ts-1/wal/instance:
uuid: "097e0db38a304bdcb2e0b46b421e65be"
format_stamp: "Formatted at 2026-09-25 15:21:57 on dist-test-slave-4bk3"
server_key: "760f70be026b84aeec490e7b7bd0c258"
server_key_iv: "58985c4becfb05a68d3e2a8febcd4e93"
server_key_version: "kuduclusterkey@0"
I20260925 15:21:57.442384 15349 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.002s
I20260925 15:21:57.459846 15366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:57.460039 15349 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:21:57.460129 15349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ts-1/wal
uuid: "097e0db38a304bdcb2e0b46b421e65be"
format_stamp: "Formatted at 2026-09-25 15:21:57 on dist-test-slave-4bk3"
server_key: "760f70be026b84aeec490e7b7bd0c258"
server_key_iv: "58985c4becfb05a68d3e2a8febcd4e93"
server_key_version: "kuduclusterkey@0"
I20260925 15:21:57.460247 15349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:21:57.484078 15349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:21:57.486682 15349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:21:57.486909 15349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:21:57.487222 15349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:21:57.487619 15349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:21:57.487703 15349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:57.487774 15349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:21:57.487828 15349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:57.495332 15349 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.194:37215
I20260925 15:21:57.495864 15349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
I20260925 15:21:57.499293 15480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.194:37215 every 8 connection(s)
I20260925 15:21:57.503882 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 15349
I20260925 15:21:57.504022 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ts-1/wal/instance
I20260925 15:21:57.504235 3919 external_mini_cluster.cc:1428] Setting key 5c255a942841ae84c663245151fae872
Sep 25 15:21:57 dist-test-slave-4bk3 krb5kdc[14655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349717, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.194@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:21:57.511520 15346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:57.500571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:48453 (local address 127.3.211.254:33937)
0925 15:21:57.500621 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:57.500623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:57.500639 (+ 16us) server_negotiation.cc:407] Connection header received
0925 15:21:57.501229 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:57.501231 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:57.501248 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:57.501279 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:57.502152 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:57.502906 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:57.504623 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:57.504915 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:57.504957 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:57.507854 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:21:57.507856 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:21:57.507856 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:21:57.507866 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:21:57.509709 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:57.510297 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:57.510298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:57.510299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:57.510364 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:57.510657 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:57.510658 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:57.510658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:57.510845 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:21:57.510893 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:57.511309 (+ 416us) server_negotiation.cc:299] Negotiation successful
0925 15:21:57.511418 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 15:21:57.512598 15483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:57.500149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:33937 (local address 127.3.211.194:48453)
0925 15:21:57.500393 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:57.500447 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:57.501090 (+ 643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:57.501469 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:57.501474 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:57.501741 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:21:57.502031 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:57.502038 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:57.503035 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:57.503036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:57.504510 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:57.504513 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:57.505052 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:57.505053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:57.505205 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:57.505874 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:21:57.505886 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:21:57.507713 (+ 1827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:21:57.509831 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:21:57.509841 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:21:57.509845 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:21:57.510194 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:21:57.510436 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:21:57.510437 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:21:57.510437 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:21:57.510581 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:21:57.510944 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:21:57.510951 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:21:57.511140 (+ 189us) client_negotiation.cc:769] Sending connection context
0925 15:21:57.512096 (+ 956us) client_negotiation.cc:240] Negotiation successful
0925 15:21:57.512297 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":107,"threads_started":1}
I20260925 15:21:57.513317 15481 heartbeater.cc:347] Connected to a master server at 127.3.211.254:33937
I20260925 15:21:57.513418 15481 heartbeater.cc:464] Registering TS with master...
I20260925 15:21:57.513616 15481 heartbeater.cc:511] Master 127.3.211.254:33937 requested a full tablet report, sending...
I20260925 15:21:57.514047 15131 ts_manager.cc:194] Registered new tserver with Master: 097e0db38a304bdcb2e0b46b421e65be (127.3.211.194:37215)
I20260925 15:21:57.514537 15131 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.194@KRBTEST.COM'} at 127.3.211.194:48453
WARNING: no policy specified for kudu/127.3.211.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab.
Entry for principal kudu/127.3.211.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab.
I20260925 15:21:57.548895 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.211.212:41087/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.195:0
--local_ip_for_outbound_sockets=127.3.211.195
--webserver_interface=127.3.211.195
--webserver_port=0
--tserver_master_addrs=127.3.211.254:33937
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab
--principal=kudu/127.3.211.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:33543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:21:57.681412 15486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:21:57.681806 15486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:21:57.681891 15486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:21:57.683799 15486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:21:57.684093 15486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:21:57.684201 15486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.195
I20260925 15:21:57.686738 15486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:33543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.211.212:41087/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab
--principal=kudu/127.3.211.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.211.195
--webserver_port=0
--tserver_master_addrs=127.3.211.254:33937
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.195
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:21:57.687299 15486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:21:57.687641 15486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:21:57.694464 15492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:21:57.695688 15486 server_base.cc:1034] running on GCE node
W20260925 15:21:57.696545 15491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:21:57.696802 15494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:21:57.697113 15486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:21:57.697400 15486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:21:57.698563 15486 hybrid_clock.cc:648] HybridClock initialized: now 1790349717698543 us; error 34 us; skew 500 ppm
Sep 25 15:21:57 dist-test-slave-4bk3 krb5kdc[14655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349717, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:21:57.702075 15486 init.cc:377] Logged in from keytab as kudu/127.3.211.195@KRBTEST.COM (short username kudu)
I20260925 15:21:57.702705 15486 webserver.cc:460] Webserver started at http://127.3.211.195:36289/ using document root <none> and password file <none>
I20260925 15:21:57.703035 15486 fs_manager.cc:359] Metadata directory not provided
I20260925 15:21:57.703154 15486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:21:57.703326 15486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:21:57 dist-test-slave-4bk3 krb5kdc[14655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349717, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.195@KRBTEST.COM for HTTP/127.3.211.212@KRBTEST.COM
I20260925 15:21:57.720467 15486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ts-2/data/instance:
uuid: "6fc566294cc5446885c67590fb879a85"
format_stamp: "Formatted at 2026-09-25 15:21:57 on dist-test-slave-4bk3"
server_key: "472b4e3171ed7e4299a4c502442fffcb"
server_key_iv: "120a5fc08680395453fbf133bb51fa85"
server_key_version: "kuduclusterkey@0"
I20260925 15:21:57.721045 15486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ts-2/wal/instance:
uuid: "6fc566294cc5446885c67590fb879a85"
format_stamp: "Formatted at 2026-09-25 15:21:57 on dist-test-slave-4bk3"
server_key: "472b4e3171ed7e4299a4c502442fffcb"
server_key_iv: "120a5fc08680395453fbf133bb51fa85"
server_key_version: "kuduclusterkey@0"
I20260925 15:21:57.724437 15486 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:21:57.744381 15501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:57.744683 15486 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 15:21:57.744791 15486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ts-2/wal
uuid: "6fc566294cc5446885c67590fb879a85"
format_stamp: "Formatted at 2026-09-25 15:21:57 on dist-test-slave-4bk3"
server_key: "472b4e3171ed7e4299a4c502442fffcb"
server_key_iv: "120a5fc08680395453fbf133bb51fa85"
server_key_version: "kuduclusterkey@0"
I20260925 15:21:57.744892 15486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:21:57.806336 15486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:21:57.809120 15486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:21:57.809360 15486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:21:57.809834 15486 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:21:57.810303 15486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:21:57.810405 15486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:57.810482 15486 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:21:57.810561 15486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:21:57.819710 15486 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.195:36685
I20260925 15:21:57.820288 15486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
I20260925 15:21:57.827803 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 15486
I20260925 15:21:57.827921 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ts-2/wal/instance
I20260925 15:21:57.828132 3919 external_mini_cluster.cc:1428] Setting key 6d01641b5bc75468b38eef286e05d5e1
I20260925 15:21:57.822968 15614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.195:36685 every 8 connection(s)
Sep 25 15:21:57 dist-test-slave-4bk3 krb5kdc[14655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349717, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.195@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:21:57.841670 15346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:57.823699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:44839 (local address 127.3.211.254:33937)
0925 15:21:57.823753 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:57.823755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:57.830598 (+ 6843us) server_negotiation.cc:407] Connection header received
0925 15:21:57.831351 (+ 753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:57.831352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:57.831366 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:57.831442 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:57.832617 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:57.833432 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:57.834954 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:57.835273 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:57.835322 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:57.838092 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:21:57.838094 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:21:57.838094 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:21:57.838102 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:21:57.839939 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:57.840531 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:57.840533 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:57.840533 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:57.840596 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:57.840881 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:57.840881 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:57.840882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:57.841021 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:21:57.841087 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:57.841471 (+ 384us) server_negotiation.cc:299] Negotiation successful
0925 15:21:57.841567 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 15:21:57.842231 15617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:57.823704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:33937 (local address 127.3.211.195:44839)
0925 15:21:57.830415 (+ 6711us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:57.830490 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:57.831214 (+ 724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:57.831629 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:57.831635 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:57.831936 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:21:57.832299 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:57.832306 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:57.833551 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:57.833553 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:57.834841 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:57.834844 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:57.835424 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:57.835425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:57.835568 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:57.836274 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:21:57.836285 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:21:57.837948 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:21:57.840059 (+ 2111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:21:57.840070 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:21:57.840073 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:21:57.840437 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:21:57.840666 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:21:57.840666 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:21:57.840667 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:21:57.840807 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:21:57.841122 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:21:57.841128 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:21:57.841309 (+ 181us) client_negotiation.cc:769] Sending connection context
0925 15:21:57.841791 (+ 482us) client_negotiation.cc:240] Negotiation successful
0925 15:21:57.841974 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6644,"thread_start_us":124,"threads_started":1}
I20260925 15:21:57.842994 15615 heartbeater.cc:347] Connected to a master server at 127.3.211.254:33937
I20260925 15:21:57.843122 15615 heartbeater.cc:464] Registering TS with master...
I20260925 15:21:57.843353 15615 heartbeater.cc:511] Master 127.3.211.254:33937 requested a full tablet report, sending...
I20260925 15:21:57.843804 15131 ts_manager.cc:194] Registered new tserver with Master: 6fc566294cc5446885c67590fb879a85 (127.3.211.195:36685)
I20260925 15:21:57.844389 15131 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.195@KRBTEST.COM'} at 127.3.211.195:44839
I20260925 15:21:57.851015 3919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:21:57.862579 15346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:57.853058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55978 (local address 127.3.211.254:33937)
0925 15:21:57.853111 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:57.853113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:57.853401 (+ 288us) server_negotiation.cc:407] Connection header received
0925 15:21:57.853525 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:57.853526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:57.853538 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:57.853586 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:21:57.854508 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:57.855277 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:57.856574 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:57.856864 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:57.856949 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:57.858855 (+ 1906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:21:57.858857 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:21:57.858857 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:21:57.858867 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:21:57.861107 (+ 2240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:57.861614 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:57.861616 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:57.861616 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:57.861699 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:21:57.861925 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:21:57.861926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:21:57.861927 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:21:57.862071 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:21:57.862156 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:57.862386 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 15:21:57.862479 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 15:21:57.866235 15131 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55978:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:21:57.869128 15131 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:21:57.881198 15629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:57.872775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:34243 (local address 127.0.0.1:37170)
0925 15:21:57.873957 (+ 1182us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:57.873967 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:57.874021 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:57.874490 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:57.874493 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:57.874495 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:21:57.874563 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:57.874565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:57.877476 (+ 2911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:57.877477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:57.878929 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:57.878932 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:57.880864 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:57.880865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:57.880957 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:57.880976 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 15:21:57.881045 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 15:21:57.881104 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1166,"thread_start_us":88,"threads_started":1}
I20260925 15:21:57.882130 15632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:57.873405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37170 (local address 127.3.211.193:34243)
0925 15:21:57.873904 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:57.873908 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:57.874300 (+ 392us) server_negotiation.cc:407] Connection header received
0925 15:21:57.874335 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:57.874337 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:57.874353 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:57.874420 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:21:57.874648 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:57.877380 (+ 2732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:57.879034 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:57.879813 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:57.879887 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:57.879953 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:57.881983 (+ 2030us) server_negotiation.cc:299] Negotiation successful
0925 15:21:57.882049 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":62,"threads_started":1}
I20260925 15:21:57.882448 15628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:57.872762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.194:37215 (local address 127.0.0.1:44898)
0925 15:21:57.873239 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:57.873283 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:57.873353 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:57.875030 (+ 1677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:57.875031 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:57.875034 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:21:57.875081 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:57.875083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:57.877076 (+ 1993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:57.877077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:57.878365 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:57.878368 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:57.882135 (+ 3767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:57.882137 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:57.882208 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:57.882262 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 15:21:57.882326 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 15:21:57.882374 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":45,"thread_start_us":80,"threads_started":1}
I20260925 15:21:57.883323 15277 tablet_service.cc:1467] Processing CreateTablet for tablet 12db593786de4fb38c5820ab80c46f8a (DEFAULT_TABLE table=test-table [id=958de1cefde24eb49fb6a88203075374]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:21:57.883713 15277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12db593786de4fb38c5820ab80c46f8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:21:57.884598 15627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:57.872992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.195:36685 (local address 127.0.0.1:52352)
0925 15:21:57.880410 (+ 7418us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:57.880423 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:57.880489 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:57.880804 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:57.880805 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:57.880807 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:21:57.880853 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:57.880855 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:57.882082 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:57.882083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:57.883198 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:57.883202 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:57.884359 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:57.884361 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:57.884425 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:57.884430 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:21:57.884484 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 15:21:57.884531 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7395,"thread_start_us":64,"threads_started":1}
I20260925 15:21:57.884832 15630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:57.873102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52352 (local address 127.3.211.195:36685)
0925 15:21:57.873488 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:57.873493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:57.880563 (+ 7070us) server_negotiation.cc:407] Connection header received
0925 15:21:57.880615 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:57.880618 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:57.880637 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:57.880716 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:21:57.880927 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:57.881989 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:57.883337 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:57.884131 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:57.884210 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:57.884290 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:57.884709 (+ 419us) server_negotiation.cc:299] Negotiation successful
0925 15:21:57.884752 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":86,"threads_started":1}
I20260925 15:21:57.885535 15549 tablet_service.cc:1467] Processing CreateTablet for tablet 12db593786de4fb38c5820ab80c46f8a (DEFAULT_TABLE table=test-table [id=958de1cefde24eb49fb6a88203075374]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:21:57.885900 15549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12db593786de4fb38c5820ab80c46f8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:21:57.882508 15631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:57.873257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44898 (local address 127.3.211.194:37215)
0925 15:21:57.874090 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:57.874094 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:57.874104 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:21:57.874143 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:57.874145 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:57.874161 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:57.874231 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:21:57.875790 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:57.876972 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:57.880016 (+ 3044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:57.881769 (+ 1753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:57.881840 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:57.881906 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:57.882380 (+ 474us) server_negotiation.cc:299] Negotiation successful
0925 15:21:57.882427 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":73,"threads_started":1}
I20260925 15:21:57.889421 15415 tablet_service.cc:1467] Processing CreateTablet for tablet 12db593786de4fb38c5820ab80c46f8a (DEFAULT_TABLE table=test-table [id=958de1cefde24eb49fb6a88203075374]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:21:57.889773 15415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12db593786de4fb38c5820ab80c46f8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:21:57.890084 15634 tablet_bootstrap.cc:492] T 12db593786de4fb38c5820ab80c46f8a P 6fc566294cc5446885c67590fb879a85: Bootstrap starting.
I20260925 15:21:57.891089 15634 tablet_bootstrap.cc:654] T 12db593786de4fb38c5820ab80c46f8a P 6fc566294cc5446885c67590fb879a85: Neither blocks nor log segments found. Creating new log.
I20260925 15:21:57.891407 15634 log.cc:824] T 12db593786de4fb38c5820ab80c46f8a P 6fc566294cc5446885c67590fb879a85: Log is configured to *not* fsync() on all Append() calls
I20260925 15:21:57.892268 15634 tablet_bootstrap.cc:492] T 12db593786de4fb38c5820ab80c46f8a P 6fc566294cc5446885c67590fb879a85: No bootstrap required, opened a new log
I20260925 15:21:57.892387 15634 ts_tablet_manager.cc:1397] T 12db593786de4fb38c5820ab80c46f8a P 6fc566294cc5446885c67590fb879a85: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:21:57.892827 15636 tablet_bootstrap.cc:492] T 12db593786de4fb38c5820ab80c46f8a P 097e0db38a304bdcb2e0b46b421e65be: Bootstrap starting.
I20260925 15:21:57.893898 15636 tablet_bootstrap.cc:654] T 12db593786de4fb38c5820ab80c46f8a P 097e0db38a304bdcb2e0b46b421e65be: Neither blocks nor log segments found. Creating new log.
I20260925 15:21:57.894295 15636 log.cc:824] T 12db593786de4fb38c5820ab80c46f8a P 097e0db38a304bdcb2e0b46b421e65be: Log is configured to *not* fsync() on all Append() calls
I20260925 15:21:57.895206 15636 tablet_bootstrap.cc:492] T 12db593786de4fb38c5820ab80c46f8a P 097e0db38a304bdcb2e0b46b421e65be: No bootstrap required, opened a new log
I20260925 15:21:57.895115 15634 raft_consensus.cc:348] T 12db593786de4fb38c5820ab80c46f8a P 6fc566294cc5446885c67590fb879a85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "097e0db38a304bdcb2e0b46b421e65be" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 37215 } } peers { permanent_uuid: "69d90e8a3597467cb07b5efd7f5a322a" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34243 } } peers { permanent_uuid: "6fc566294cc5446885c67590fb879a85" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 36685 } }
I20260925 15:21:57.895279 15636 ts_tablet_manager.cc:1397] T 12db593786de4fb38c5820ab80c46f8a P 097e0db38a304bdcb2e0b46b421e65be: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 15:21:57.895303 15634 raft_consensus.cc:374] T 12db593786de4fb38c5820ab80c46f8a P 6fc566294cc5446885c67590fb879a85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:21:57.895339 15634 raft_consensus.cc:729] T 12db593786de4fb38c5820ab80c46f8a P 6fc566294cc5446885c67590fb879a85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fc566294cc5446885c67590fb879a85, State: Initialized, Role: FOLLOWER
I20260925 15:21:57.895486 15634 consensus_queue.cc:260] T 12db593786de4fb38c5820ab80c46f8a P 6fc566294cc5446885c67590fb879a85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "097e0db38a304bdcb2e0b46b421e65be" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 37215 } } peers { permanent_uuid: "69d90e8a3597467cb07b5efd7f5a322a" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34243 } } peers { permanent_uuid: "6fc566294cc5446885c67590fb879a85" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 36685 } }
I20260925 15:21:57.895877 15634 ts_tablet_manager.cc:1428] T 12db593786de4fb38c5820ab80c46f8a P 6fc566294cc5446885c67590fb879a85: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:21:57.896018 15615 heartbeater.cc:503] Master 127.3.211.254:33937 was elected leader, sending a full tablet report...
I20260925 15:21:57.898518 15636 raft_consensus.cc:348] T 12db593786de4fb38c5820ab80c46f8a P 097e0db38a304bdcb2e0b46b421e65be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "097e0db38a304bdcb2e0b46b421e65be" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 37215 } } peers { permanent_uuid: "69d90e8a3597467cb07b5efd7f5a322a" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34243 } } peers { permanent_uuid: "6fc566294cc5446885c67590fb879a85" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 36685 } }
I20260925 15:21:57.898752 15636 raft_consensus.cc:374] T 12db593786de4fb38c5820ab80c46f8a P 097e0db38a304bdcb2e0b46b421e65be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:21:57.898799 15636 raft_consensus.cc:729] T 12db593786de4fb38c5820ab80c46f8a P 097e0db38a304bdcb2e0b46b421e65be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 097e0db38a304bdcb2e0b46b421e65be, State: Initialized, Role: FOLLOWER
I20260925 15:21:57.899025 15636 consensus_queue.cc:260] T 12db593786de4fb38c5820ab80c46f8a P 097e0db38a304bdcb2e0b46b421e65be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "097e0db38a304bdcb2e0b46b421e65be" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 37215 } } peers { permanent_uuid: "69d90e8a3597467cb07b5efd7f5a322a" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34243 } } peers { permanent_uuid: "6fc566294cc5446885c67590fb879a85" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 36685 } }
I20260925 15:21:57.899405 15636 ts_tablet_manager.cc:1428] T 12db593786de4fb38c5820ab80c46f8a P 097e0db38a304bdcb2e0b46b421e65be: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260925 15:21:57.900524 15633 tablet_bootstrap.cc:492] T 12db593786de4fb38c5820ab80c46f8a P 69d90e8a3597467cb07b5efd7f5a322a: Bootstrap starting.
I20260925 15:21:57.900934 15481 heartbeater.cc:503] Master 127.3.211.254:33937 was elected leader, sending a full tablet report...
I20260925 15:21:57.901489 15633 tablet_bootstrap.cc:654] T 12db593786de4fb38c5820ab80c46f8a P 69d90e8a3597467cb07b5efd7f5a322a: Neither blocks nor log segments found. Creating new log.
I20260925 15:21:57.901921 15633 log.cc:824] T 12db593786de4fb38c5820ab80c46f8a P 69d90e8a3597467cb07b5efd7f5a322a: Log is configured to *not* fsync() on all Append() calls
I20260925 15:21:57.902779 15633 tablet_bootstrap.cc:492] T 12db593786de4fb38c5820ab80c46f8a P 69d90e8a3597467cb07b5efd7f5a322a: No bootstrap required, opened a new log
I20260925 15:21:57.902874 15633 ts_tablet_manager.cc:1397] T 12db593786de4fb38c5820ab80c46f8a P 69d90e8a3597467cb07b5efd7f5a322a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:21:57.906057 15633 raft_consensus.cc:348] T 12db593786de4fb38c5820ab80c46f8a P 69d90e8a3597467cb07b5efd7f5a322a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "097e0db38a304bdcb2e0b46b421e65be" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 37215 } } peers { permanent_uuid: "69d90e8a3597467cb07b5efd7f5a322a" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34243 } } peers { permanent_uuid: "6fc566294cc5446885c67590fb879a85" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 36685 } }
I20260925 15:21:57.906296 15633 raft_consensus.cc:374] T 12db593786de4fb38c5820ab80c46f8a P 69d90e8a3597467cb07b5efd7f5a322a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:21:57.906349 15633 raft_consensus.cc:729] T 12db593786de4fb38c5820ab80c46f8a P 69d90e8a3597467cb07b5efd7f5a322a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69d90e8a3597467cb07b5efd7f5a322a, State: Initialized, Role: FOLLOWER
I20260925 15:21:57.906525 15633 consensus_queue.cc:260] T 12db593786de4fb38c5820ab80c46f8a P 69d90e8a3597467cb07b5efd7f5a322a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "097e0db38a304bdcb2e0b46b421e65be" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 37215 } } peers { permanent_uuid: "69d90e8a3597467cb07b5efd7f5a322a" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34243 } } peers { permanent_uuid: "6fc566294cc5446885c67590fb879a85" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 36685 } }
I20260925 15:21:57.906957 15343 heartbeater.cc:503] Master 127.3.211.254:33937 was elected leader, sending a full tablet report...
I20260925 15:21:57.907208 15633 ts_tablet_manager.cc:1428] T 12db593786de4fb38c5820ab80c46f8a P 69d90e8a3597467cb07b5efd7f5a322a: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260925 15:21:57.919929 15641 raft_consensus.cc:482] T 12db593786de4fb38c5820ab80c46f8a P 69d90e8a3597467cb07b5efd7f5a322a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:21:57.920091 15641 raft_consensus.cc:504] T 12db593786de4fb38c5820ab80c46f8a P 69d90e8a3597467cb07b5efd7f5a322a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "097e0db38a304bdcb2e0b46b421e65be" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 37215 } } peers { permanent_uuid: "69d90e8a3597467cb07b5efd7f5a322a" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34243 } } peers { permanent_uuid: "6fc566294cc5446885c67590fb879a85" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 36685 } }
I20260925 15:21:57.920538 15641 leader_election.cc:288] T 12db593786de4fb38c5820ab80c46f8a P 69d90e8a3597467cb07b5efd7f5a322a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 097e0db38a304bdcb2e0b46b421e65be (127.3.211.194:37215), 6fc566294cc5446885c67590fb879a85 (127.3.211.195:36685)
I20260925 15:21:57.926426 15631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:57.921168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:43303 (local address 127.3.211.194:37215)
0925 15:21:57.921263 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:57.921265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:57.922695 (+ 1430us) server_negotiation.cc:407] Connection header received
0925 15:21:57.922809 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:57.922811 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:57.922820 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:57.922849 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:21:57.923082 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:57.923851 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:57.925443 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:57.926032 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:57.926082 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:57.926091 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:57.926305 (+ 214us) server_negotiation.cc:299] Negotiation successful
0925 15:21:57.926346 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 15:21:57.926913 15643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:57.920741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.194:37215 (local address 127.3.211.193:43303)
0925 15:21:57.922597 (+ 1856us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:57.922612 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:57.922731 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:57.922916 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:57.922917 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:57.922928 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:21:57.923010 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:57.923012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:57.923987 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:57.923988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:57.925333 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:57.925335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:57.926151 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:57.926152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:57.926224 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:57.926229 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:21:57.926779 (+ 550us) client_negotiation.cc:240] Negotiation successful
0925 15:21:57.926831 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1828,"thread_start_us":121,"threads_started":1}
I20260925 15:21:57.927418 15432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12db593786de4fb38c5820ab80c46f8a" candidate_uuid: "69d90e8a3597467cb07b5efd7f5a322a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "097e0db38a304bdcb2e0b46b421e65be" is_pre_election: true
I20260925 15:21:57.927606 15432 raft_consensus.cc:2454] T 12db593786de4fb38c5820ab80c46f8a P 097e0db38a304bdcb2e0b46b421e65be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69d90e8a3597467cb07b5efd7f5a322a in term 0.
I20260925 15:21:57.927896 15228 leader_election.cc:302] T 12db593786de4fb38c5820ab80c46f8a P 69d90e8a3597467cb07b5efd7f5a322a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 097e0db38a304bdcb2e0b46b421e65be, 69d90e8a3597467cb07b5efd7f5a322a; no voters:
I20260925 15:21:57.928082 15630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:57.921216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:49711 (local address 127.3.211.195:36685)
0925 15:21:57.921297 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:57.921299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:57.924616 (+ 3317us) server_negotiation.cc:407] Connection header received
0925 15:21:57.924767 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:57.924769 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:57.924779 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:57.924810 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:21:57.925027 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:57.925759 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:57.927170 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:57.927669 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:57.927738 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:57.927749 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:57.927968 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 15:21:57.928015 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 15:21:57.928093 15641 raft_consensus.cc:2781] T 12db593786de4fb38c5820ab80c46f8a P 69d90e8a3597467cb07b5efd7f5a322a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:21:57.928362 15641 raft_consensus.cc:482] T 12db593786de4fb38c5820ab80c46f8a P 69d90e8a3597467cb07b5efd7f5a322a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:21:57.928435 15642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:57.920960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.195:36685 (local address 127.3.211.193:49711)
0925 15:21:57.924511 (+ 3551us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:57.924525 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:21:57.924669 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:21:57.924877 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:21:57.924878 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:21:57.924880 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:21:57.924957 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:57.924958 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:57.925884 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:57.925885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:57.927056 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:21:57.927059 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:21:57.927814 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:21:57.927815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:21:57.927891 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:57.927895 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:21:57.928221 (+ 326us) client_negotiation.cc:240] Negotiation successful
0925 15:21:57.928269 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3508,"thread_start_us":56,"threads_started":1}
I20260925 15:21:57.928450 15641 raft_consensus.cc:3037] T 12db593786de4fb38c5820ab80c46f8a P 69d90e8a3597467cb07b5efd7f5a322a [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:21:57.928923 15553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12db593786de4fb38c5820ab80c46f8a" candidate_uuid: "69d90e8a3597467cb07b5efd7f5a322a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fc566294cc5446885c67590fb879a85" is_pre_election: true
I20260925 15:21:57.929076 15553 raft_consensus.cc:2454] T 12db593786de4fb38c5820ab80c46f8a P 6fc566294cc5446885c67590fb879a85 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69d90e8a3597467cb07b5efd7f5a322a in term 0.
I20260925 15:21:57.929772 15641 raft_consensus.cc:504] T 12db593786de4fb38c5820ab80c46f8a P 69d90e8a3597467cb07b5efd7f5a322a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "097e0db38a304bdcb2e0b46b421e65be" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 37215 } } peers { permanent_uuid: "69d90e8a3597467cb07b5efd7f5a322a" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34243 } } peers { permanent_uuid: "6fc566294cc5446885c67590fb879a85" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 36685 } }
I20260925 15:21:57.929984 15641 leader_election.cc:288] T 12db593786de4fb38c5820ab80c46f8a P 69d90e8a3597467cb07b5efd7f5a322a [CANDIDATE]: Term 1 election: Requested vote from peers 097e0db38a304bdcb2e0b46b421e65be (127.3.211.194:37215), 6fc566294cc5446885c67590fb879a85 (127.3.211.195:36685)
I20260925 15:21:57.930202 15432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12db593786de4fb38c5820ab80c46f8a" candidate_uuid: "69d90e8a3597467cb07b5efd7f5a322a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "097e0db38a304bdcb2e0b46b421e65be"
I20260925 15:21:57.930203 15553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12db593786de4fb38c5820ab80c46f8a" candidate_uuid: "69d90e8a3597467cb07b5efd7f5a322a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6fc566294cc5446885c67590fb879a85"
I20260925 15:21:57.930289 15432 raft_consensus.cc:3037] T 12db593786de4fb38c5820ab80c46f8a P 097e0db38a304bdcb2e0b46b421e65be [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:21:57.930287 15553 raft_consensus.cc:3037] T 12db593786de4fb38c5820ab80c46f8a P 6fc566294cc5446885c67590fb879a85 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:21:57.931205 15432 raft_consensus.cc:2454] T 12db593786de4fb38c5820ab80c46f8a P 097e0db38a304bdcb2e0b46b421e65be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69d90e8a3597467cb07b5efd7f5a322a in term 1.
I20260925 15:21:57.931214 15553 raft_consensus.cc:2454] T 12db593786de4fb38c5820ab80c46f8a P 6fc566294cc5446885c67590fb879a85 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69d90e8a3597467cb07b5efd7f5a322a in term 1.
I20260925 15:21:57.931624 15228 leader_election.cc:302] T 12db593786de4fb38c5820ab80c46f8a P 69d90e8a3597467cb07b5efd7f5a322a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69d90e8a3597467cb07b5efd7f5a322a, 6fc566294cc5446885c67590fb879a85; no voters:
I20260925 15:21:57.931850 15641 raft_consensus.cc:2781] T 12db593786de4fb38c5820ab80c46f8a P 69d90e8a3597467cb07b5efd7f5a322a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:21:57.932030 15641 raft_consensus.cc:686] T 12db593786de4fb38c5820ab80c46f8a P 69d90e8a3597467cb07b5efd7f5a322a [term 1 LEADER]: Becoming Leader. State: Replica: 69d90e8a3597467cb07b5efd7f5a322a, State: Running, Role: LEADER
I20260925 15:21:57.932179 15641 consensus_queue.cc:237] T 12db593786de4fb38c5820ab80c46f8a P 69d90e8a3597467cb07b5efd7f5a322a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "097e0db38a304bdcb2e0b46b421e65be" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 37215 } } peers { permanent_uuid: "69d90e8a3597467cb07b5efd7f5a322a" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34243 } } peers { permanent_uuid: "6fc566294cc5446885c67590fb879a85" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 36685 } }
I20260925 15:21:57.933239 15130 catalog_manager.cc:5495] T 12db593786de4fb38c5820ab80c46f8a P 69d90e8a3597467cb07b5efd7f5a322a reported cstate change: term changed from 0 to 1, leader changed from <none> to 69d90e8a3597467cb07b5efd7f5a322a (127.3.211.193). New cstate: current_term: 1 leader_uuid: "69d90e8a3597467cb07b5efd7f5a322a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "097e0db38a304bdcb2e0b46b421e65be" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 37215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69d90e8a3597467cb07b5efd7f5a322a" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34243 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6fc566294cc5446885c67590fb879a85" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 36685 } health_report { overall_health: UNKNOWN } } }
W20260925 15:21:57.949453 15344 tablet.cc:2369] T 12db593786de4fb38c5820ab80c46f8a P 69d90e8a3597467cb07b5efd7f5a322a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:21:57.954432 15632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:57.950499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37180 (local address 127.3.211.193:34243)
0925 15:21:57.950702 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 15:21:57.950704 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:21:57.950712 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:21:57.950764 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:21:57.950765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:21:57.950786 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:21:57.950868 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:21:57.951057 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:57.951844 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:57.953156 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:21:57.953531 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:21:57.953605 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:21:57.953836 (+ 231us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:21:57.953970 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:21:57.954048 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 15:21:57.954305 (+ 257us) server_negotiation.cc:299] Negotiation successful
0925 15:21:57.954331 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169}
I20260925 15:21:57.961858 15432 raft_consensus.cc:1261] T 12db593786de4fb38c5820ab80c46f8a P 097e0db38a304bdcb2e0b46b421e65be [term 1 FOLLOWER]: Refusing update from remote peer 69d90e8a3597467cb07b5efd7f5a322a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:21:57.961907 15553 raft_consensus.cc:1261] T 12db593786de4fb38c5820ab80c46f8a P 6fc566294cc5446885c67590fb879a85 [term 1 FOLLOWER]: Refusing update from remote peer 69d90e8a3597467cb07b5efd7f5a322a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:21:57.962313 15644 consensus_queue.cc:1035] T 12db593786de4fb38c5820ab80c46f8a P 69d90e8a3597467cb07b5efd7f5a322a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6fc566294cc5446885c67590fb879a85" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 36685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:21:57.962499 15644 consensus_queue.cc:1035] T 12db593786de4fb38c5820ab80c46f8a P 69d90e8a3597467cb07b5efd7f5a322a [LEADER]: Connected to new peer: Peer: permanent_uuid: "097e0db38a304bdcb2e0b46b421e65be" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 37215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:21:57.965969 15647 mvcc.cc:205] Tried to move back new op lower bound from 7333272444767207424 to 7333272444651847680. Current Snapshot: MvccSnapshot[applied={T|T < 7333272444767207424}]
I20260925 15:21:57.969233 15651 mvcc.cc:205] Tried to move back new op lower bound from 7333272444767207424 to 7333272444651847680. Current Snapshot: MvccSnapshot[applied={T|T < 7333272444767207424 or (T in {7333272444767207424})}]
I20260925 15:21:57.969866 15648 mvcc.cc:205] Tried to move back new op lower bound from 7333272444767207424 to 7333272444651847680. Current Snapshot: MvccSnapshot[applied={T|T < 7333272444767207424}]
Sep 25 15:21:57 dist-test-slave-4bk3 krb5kdc[14655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349717, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.254@KRBTEST.COM for HTTP/127.3.211.212@KRBTEST.COM
I20260925 15:21:57.970901 15130 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55978:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:21:57.971002 15130 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55978:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:21:57.971906 15130 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8452a69e9a5b4ce5b33e0ade56b99a18: Sending DeleteTablet for 3 replicas of tablet 12db593786de4fb38c5820ab80c46f8a
I20260925 15:21:57.972280 15277 tablet_service.cc:1514] Processing DeleteTablet for tablet 12db593786de4fb38c5820ab80c46f8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:21:57 UTC) from {username='kudu', principal='kudu/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:37170
I20260925 15:21:57.972842 15415 tablet_service.cc:1514] Processing DeleteTablet for tablet 12db593786de4fb38c5820ab80c46f8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:21:57 UTC) from {username='kudu', principal='kudu/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:44898
I20260925 15:21:57.973086 15549 tablet_service.cc:1514] Processing DeleteTablet for tablet 12db593786de4fb38c5820ab80c46f8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:21:57 UTC) from {username='kudu', principal='kudu/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:52352
I20260925 15:21:57.973886 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 15212
I20260925 15:21:57.976181 15657 tablet_replica.cc:330] T 12db593786de4fb38c5820ab80c46f8a P 097e0db38a304bdcb2e0b46b421e65be: stopping tablet replica
I20260925 15:21:57.976401 15657 raft_consensus.cc:2229] T 12db593786de4fb38c5820ab80c46f8a P 097e0db38a304bdcb2e0b46b421e65be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:21:57.976613 15658 tablet_replica.cc:330] T 12db593786de4fb38c5820ab80c46f8a P 6fc566294cc5446885c67590fb879a85: stopping tablet replica
I20260925 15:21:57.976784 15658 raft_consensus.cc:2229] T 12db593786de4fb38c5820ab80c46f8a P 6fc566294cc5446885c67590fb879a85 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:21:57.976976 15658 raft_consensus.cc:2258] T 12db593786de4fb38c5820ab80c46f8a P 6fc566294cc5446885c67590fb879a85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:21:57.977128 15657 raft_consensus.cc:2258] T 12db593786de4fb38c5820ab80c46f8a P 097e0db38a304bdcb2e0b46b421e65be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:21:57.979317 15657 ts_tablet_manager.cc:1905] T 12db593786de4fb38c5820ab80c46f8a P 097e0db38a304bdcb2e0b46b421e65be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:21:57.981061 15657 ts_tablet_manager.cc:1918] T 12db593786de4fb38c5820ab80c46f8a P 097e0db38a304bdcb2e0b46b421e65be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:21:57.981226 15658 ts_tablet_manager.cc:1905] T 12db593786de4fb38c5820ab80c46f8a P 6fc566294cc5446885c67590fb879a85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:21:57.982678 15657 log.cc:1196] T 12db593786de4fb38c5820ab80c46f8a P 097e0db38a304bdcb2e0b46b421e65be: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ts-1/wal/wals/12db593786de4fb38c5820ab80c46f8a
I20260925 15:21:57.983004 15657 ts_tablet_manager.cc:1939] T 12db593786de4fb38c5820ab80c46f8a P 097e0db38a304bdcb2e0b46b421e65be: Deleting consensus metadata
I20260925 15:21:57.983122 15658 ts_tablet_manager.cc:1918] T 12db593786de4fb38c5820ab80c46f8a P 6fc566294cc5446885c67590fb879a85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:21:57.983219 15658 log.cc:1196] T 12db593786de4fb38c5820ab80c46f8a P 6fc566294cc5446885c67590fb879a85: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790349610595785-3919-0/minicluster-data/ts-2/wal/wals/12db593786de4fb38c5820ab80c46f8a
W20260925 15:21:57.983407 15116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.211.193:34243 (error 108)
I20260925 15:21:57.983734 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 15349
I20260925 15:21:57.987696 15658 ts_tablet_manager.cc:1939] T 12db593786de4fb38c5820ab80c46f8a P 6fc566294cc5446885c67590fb879a85: Deleting consensus metadata
I20260925 15:21:57.992138 15115 catalog_manager.cc:4841] TS 6fc566294cc5446885c67590fb879a85 (127.3.211.195:36685): tablet 12db593786de4fb38c5820ab80c46f8a (table test-table [id=958de1cefde24eb49fb6a88203075374]) successfully deleted
I20260925 15:21:57.992910 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 15486
I20260925 15:21:57.997289 15627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:57.997099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:34243 (local address 127.0.0.1:37194)
0925 15:21:57.997163 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:57.997224 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.193:34243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 15:21:57.997296 15628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:21:57.997129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.194:37215 (local address 127.0.0.1:44904)
0925 15:21:57.997181 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 15:21:57.997225 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.194:37215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 15:21:57.997481 15115 catalog_manager.cc:4584] TS 097e0db38a304bdcb2e0b46b421e65be (127.3.211.194:37215): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 12db593786de4fb38c5820ab80c46f8a: Network error: Client connection negotiation failed: client connection to 127.3.211.194:37215: connect: Connection refused (error 111)
I20260925 15:21:58.001624 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 15099
I20260925 15:21:58.010941 3919 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 15:21:58.356681 3919 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 15:21:58.356781 3919 mini_ranger.cc:66] Stopping Ranger...
I20260925 15:21:58.702762 3919 mini_ranger.cc:68] Stopped Ranger
2026-09-25 15:21:58.703 UTC [14685] LOG: received smart shutdown request
2026-09-25 15:21:58.707 UTC [14685] LOG: background worker "logical replication launcher" (PID 14694) exited with exit code 1
2026-09-25 15:21:58.707 UTC [14689] LOG: shutting down
2026-09-25 15:21:58.907 UTC [14685] LOG: database system is shut down
2026-09-25T15:21:58Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48967 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:21:59 dist-test-slave-4bk3 krb5kdc[15668](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:21:59 dist-test-slave-4bk3 krb5kdc[15668](info): set up 2 sockets
Sep 25 15:21:59 dist-test-slave-4bk3 krb5kdc[15668](info): commencing operation
krb5kdc: starting...
W20260925 15:22:00.128260 3919 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Sep 25 15:22:00 dist-test-slave-4bk3 krb5kdc[15668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:22:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:22:00Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.211.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.211.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.211.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/rangeradmin_127.3.211.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.211.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/rangeradmin_127.3.211.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.211.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.211.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.211.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/rangerlookup_127.3.211.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.211.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/rangerlookup_127.3.211.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.211.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.211.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.211.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/HTTP_127.3.211.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.211.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/HTTP_127.3.211.212@KRBTEST.COM.keytab.
I20260925 15:22:00.312075 3919 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskAzNr2f/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/postgres -l logfile start
2026-09-25 15:22:03.212 UTC [15698] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 15:22:03.212 UTC [15698] LOG: listening on IPv4 address "127.3.211.212", port 47723
2026-09-25 15:22:03.216 UTC [15698] LOG: listening on Unix socket "/tmp/.s.PGSQL.47723"
2026-09-25 15:22:03.254 UTC [15701] LOG: database system was shut down at 2026-09-25 15:22:01 UTC
2026-09-25 15:22:03.258 UTC [15698] LOG: database system is ready to accept connections
I20260925 15:22:04.220191 3919 mini_postgres.cc:96] Postgres bound to 47723
2026-09-25 15:22:04.226 UTC [15709] FATAL: database "slave" does not exist
127.3.211.212:47723 - accepting connections
I20260925 15:22:04.227301 3919 mini_ranger.cc:161] Starting Ranger...
I20260925 15:22:04.243518 3919 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 15:22:04.812443 3919 mini_ranger.cc:87] Created ranger Postgres database
I20260925 15:22:04.812529 3919 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ranger-admin
2026-09-25 15:22:05,101 [I] DB FLAVOR :POSTGRES
2026-09-25 15:22:05,102 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 15:22:05,102 [I] Checking connection..
2026-09-25 15:22:05,102 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNr2f/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNr2f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.211.212:47723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 15:22:05,382 [I] Checking connection passed.
2026-09-25 15:22:05,382 [I] --------- Verifying version history table ---------
2026-09-25 15:22:05,382 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNr2f/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNr2f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.211.212:47723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 15:22:05,642 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 15:22:05,642 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNr2f/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNr2f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.211.212:47723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 15:22:05,928 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 15:22:05,928 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 15:22:05,928 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNr2f/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNr2f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.211.212:47723/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAzNr2f/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 15:22:06.192 UTC [15773] WARNING: there is no transaction in progress
2026-09-25 15:22:06,199 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 15:22:06,199 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNr2f/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNr2f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.211.212:47723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 15:22:06,484 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 15:22:06,485 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 15:22:06,485 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNr2f/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNr2f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.211.212:47723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 15:22:06,734 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNr2f/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNr2f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.211.212:47723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 15:22:06,982 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNr2f/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNr2f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.211.212:47723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-4bk3.c.gcp-upstream.internal','N') ;"
2026-09-25 15:22:07,243 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNr2f/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNr2f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.211.212:47723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 15:22:07,506 [I] Table x_portal_user does not exist in database ranger
2026-09-25 15:22:07,506 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNr2f/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNr2f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.211.212:47723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 15:22:07,765 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 15:22:07,765 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNr2f/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNr2f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.211.212:47723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 15:22:08,018 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNr2f/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNr2f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.211.212:47723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 15:22:08,279 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 15:22:08,279 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNr2f/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNr2f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.211.212:47723/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAzNr2f/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 15:22:09.378 UTC [15881] WARNING: there is no transaction in progress
2026-09-25 15:22:09.396 UTC [15881] WARNING: there is no transaction in progress
2026-09-25 15:22:09.415 UTC [15881] WARNING: there is no transaction in progress
2026-09-25 15:22:09.433 UTC [15881] WARNING: there is no transaction in progress
2026-09-25 15:22:09.452 UTC [15881] WARNING: there is no transaction in progress
2026-09-25 15:22:09.471 UTC [15881] WARNING: there is no transaction in progress
2026-09-25 15:22:09.534 UTC [15881] WARNING: there is no transaction in progress
2026-09-25 15:22:09.542 UTC [15881] WARNING: there is no transaction in progress
2026-09-25 15:22:09.560 UTC [15881] WARNING: there is no transaction in progress
2026-09-25 15:22:09.573 UTC [15881] WARNING: there is no transaction in progress
2026-09-25 15:22:09.585 UTC [15881] WARNING: there is no transaction in progress
2026-09-25 15:22:09.600 UTC [15881] WARNING: there is no transaction in progress
2026-09-25 15:22:09.611 UTC [15881] WARNING: there is no transaction in progress
2026-09-25 15:22:09.619 UTC [15881] WARNING: there is no transaction in progress
2026-09-25 15:22:09.628 UTC [15881] WARNING: there is no transaction in progress
2026-09-25 15:22:09.638 UTC [15881] WARNING: there is no transaction in progress
2026-09-25 15:22:10,425 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 15:22:10,426 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNr2f/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNr2f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.211.212:47723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 15:22:10,675 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 15:22:10,675 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNr2f/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNr2f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.211.212:47723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 15:22:10,942 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 15:22:10,943 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNr2f/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNr2f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.211.212:47723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 15:22:11,192 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 15:22:11,192 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNr2f/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNr2f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.211.212:47723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 15:22:11,450 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNr2f/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNr2f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.211.212:47723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 15:22:11,701 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNr2f/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNr2f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.211.212:47723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-4bk3.c.gcp-upstream.internal';"
2026-09-25 15:22:11,946 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 15:22:11,946 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNr2f/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNr2f/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.211.212:47723/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 15:22:12,218 [I] DB_PATCHES have already been applied
I20260925 15:22:12.223798 3919 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ranger-admin:/tmp/dist-test-taskAzNr2f/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskAzNr2f/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskAzNr2f/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNr2f/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 15:22:12.223887 3919 mini_ranger.cc:193] Using host: 127.3.211.212
I20260925 15:22:12.227303 3919 mini_ranger.cc:227] Ranger admin URL: http://127.3.211.212:44155
Sep 25, 2026 3:22:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 3:22:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 3:22:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 3:22:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 3:22:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 3:22:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 3:22:13 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 15:22:13.363963 3919 mini_ranger.cc:160] Time spent starting Ranger: real 9.137s user 0.000s sys 0.009s
Sep 25, 2026 3:22:36 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 3:22:36 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 25, 2026 3:22:36 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 3:22:36 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 3:22:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 3:22:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 3:22:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 3:22:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 3:22:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 3:22:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 3:22:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 3:22:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 3:22:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 3:22:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 3:22:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 3:22:36 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 3:22:36 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 3:22:37 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 15:22:38.116299 3919 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.211.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.211.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.211.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/rangerkms_127.3.211.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.211.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/rangerkms_127.3.211.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.211.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.211.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.211.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/HTTP_127.3.211.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.211.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/HTTP_127.3.211.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 15:22:38 dist-test-slave-4bk3 krb5kdc[15668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349758, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 15:22:38.207340 3919 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 15:22:38.218235 3919 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 15:22:39.369370 3919 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 15:22:39.369442 3919 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ranger-kms
I20260925 15:22:39.369455 3919 mini_ranger_kms.cc:227] Using postgres at 127.3.211.212:47723
2026-09-25 15:22:39,554 [I] DB FLAVOR :POSTGRES
2026-09-25 15:22:39,554 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 15:22:39,554 [I] Checking connection
2026-09-25 15:22:39,554 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNr2f/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.211.212:47723/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 15:22:39,815 [I] connection success
2026-09-25 15:22:39,815 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 15:22:39,815 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 15:22:39,815 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNr2f/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.211.212:47723/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 15:22:40,071 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 15:22:40,071 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 15:22:40,072 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 15:22:40,072 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskAzNr2f/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.211.212:47723/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 15:22:40,363 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 15:22:40.985392 3919 mini_ranger_kms.cc:310] Created kms service
I20260925 15:22:41.188406 3919 mini_ranger_kms.cc:326] Created kudu user
I20260925 15:22:41.245869 3919 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 15:22:41 dist-test-slave-4bk3 krb5kdc[15668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349758, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.211.212@KRBTEST.COM
I20260925 15:22:41.671339 3919 mini_ranger_kms.cc:384] Added ranger policy
I20260925 15:22:41.671583 3919 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ranger-kms:/tmp/dist-test-taskAzNr2f/build/release/bin/postgresql.jar:/tmp/dist-test-taskAzNr2f/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskAzNr2f/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 15:22:41.671607 3919 mini_ranger_kms.cc:240] Using host: 127.3.211.212
I20260925 15:22:41.675477 3919 mini_ranger_kms.cc:276] Ranger KMS PID: 16062
I20260925 15:22:41.675571 3919 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.211.212:57279
Sep 25, 2026 3:22:42 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 3:22:42 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 3:22:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 3:22:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 3:22:43 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 3:22:43 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 15:22:43.988139 3919 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.781s user 0.000s sys 0.164s
I20260925 15:22:43.988251 3919 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 15:22:43.988303 3919 mini_ranger_kms.cc:401] 127.3.211.212:57279/kms/v1/keys
Sep 25 15:22:44 dist-test-slave-4bk3 krb5kdc[15668](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790349764, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.211.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 15:22:44 dist-test-slave-4bk3 krb5kdc[15668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349764, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.211.212@KRBTEST.COM for HTTP/127.3.211.212@KRBTEST.COM
Sep 25 15:22:44 dist-test-slave-4bk3 krb5kdc[15668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349764, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.211.212@KRBTEST.COM for HTTP/127.3.211.212@KRBTEST.COM
Sep 25 15:22:47 dist-test-slave-4bk3 krb5kdc[15668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.211.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
I20260925 15:22:47.392071 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:34085
--webserver_interface=127.3.211.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:41295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.211.254:34085
--ranger_config_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.211.212:57279/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:22:47.476325 16112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:22:47.476619 16112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:22:47.476644 16112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:22:47.477972 16112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:22:47.478045 16112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:22:47.478061 16112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:22:47.478075 16112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:22:47.478086 16112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:22:47.479990 16112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:41295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.211.212:57279/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.211.254:34085
--ranger_config_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:34085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.211.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:22:47.480244 16112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:22:47.480535 16112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:22:47.485865 16118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:47.485870 16120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:22:47.486156 16112 server_base.cc:1034] running on GCE node
W20260925 15:22:47.485870 16117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:22:47.486721 16112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:22:47.487112 16112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:22:47.488406 16112 hybrid_clock.cc:648] HybridClock initialized: now 1790349767488379 us; error 31 us; skew 500 ppm
Sep 25 15:22:47 dist-test-slave-4bk3 krb5kdc[15668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349767, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:22:47.491261 16112 init.cc:377] Logged in from keytab as kudu/127.3.211.254@KRBTEST.COM (short username kudu)
I20260925 15:22:47.491787 16112 webserver.cc:460] Webserver started at http://127.3.211.254:43199/ using document root <none> and password file <none>
I20260925 15:22:47.492048 16112 fs_manager.cc:359] Metadata directory not provided
I20260925 15:22:47.492116 16112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:22:47.492280 16112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:22:47 dist-test-slave-4bk3 krb5kdc[15668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349767, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.254@KRBTEST.COM for HTTP/127.3.211.212@KRBTEST.COM
I20260925 15:22:47.544037 16112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/master-0/data/instance:
uuid: "b9426d03e35647be9b6fea84c8c8ee75"
format_stamp: "Formatted at 2026-09-25 15:22:47 on dist-test-slave-4bk3"
server_key: "d08a053a083ce6149eb83f4686ad37a2"
server_key_iv: "16aab3315534929d909e7dbcaaa506a5"
server_key_version: "kuduclusterkey@0"
I20260925 15:22:47.544529 16112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/master-0/wal/instance:
uuid: "b9426d03e35647be9b6fea84c8c8ee75"
format_stamp: "Formatted at 2026-09-25 15:22:47 on dist-test-slave-4bk3"
server_key: "d08a053a083ce6149eb83f4686ad37a2"
server_key_iv: "16aab3315534929d909e7dbcaaa506a5"
server_key_version: "kuduclusterkey@0"
I20260925 15:22:47.546021 16112 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:22:47.563730 16127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:47.564013 16112 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:22:47.564132 16112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/master-0/wal
uuid: "b9426d03e35647be9b6fea84c8c8ee75"
format_stamp: "Formatted at 2026-09-25 15:22:47 on dist-test-slave-4bk3"
server_key: "d08a053a083ce6149eb83f4686ad37a2"
server_key_iv: "16aab3315534929d909e7dbcaaa506a5"
server_key_version: "kuduclusterkey@0"
I20260925 15:22:47.564247 16112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:22:47.605891 16112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:22:47.608520 16112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:22:47.608654 16112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:22:47.613476 16112 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.254:34085
I20260925 15:22:47.613523 16179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.254:34085 every 8 connection(s)
I20260925 15:22:47.613914 16112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
I20260925 15:22:47.614611 16180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:22:47.617103 16180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9426d03e35647be9b6fea84c8c8ee75: Bootstrap starting.
I20260925 15:22:47.617691 16180 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9426d03e35647be9b6fea84c8c8ee75: Neither blocks nor log segments found. Creating new log.
I20260925 15:22:47.617942 16180 log.cc:824] T 00000000000000000000000000000000 P b9426d03e35647be9b6fea84c8c8ee75: Log is configured to *not* fsync() on all Append() calls
I20260925 15:22:47.618759 16180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9426d03e35647be9b6fea84c8c8ee75: No bootstrap required, opened a new log
I20260925 15:22:47.619537 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 16112
I20260925 15:22:47.619643 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/master-0/wal/instance
I20260925 15:22:47.619854 3919 external_mini_cluster.cc:1428] Setting key faa02f102216cc3eb492156cac871d88
I20260925 15:22:47.621311 16180 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9426d03e35647be9b6fea84c8c8ee75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9426d03e35647be9b6fea84c8c8ee75" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 34085 } }
I20260925 15:22:47.621503 16180 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9426d03e35647be9b6fea84c8c8ee75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:22:47.621567 16180 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9426d03e35647be9b6fea84c8c8ee75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9426d03e35647be9b6fea84c8c8ee75, State: Initialized, Role: FOLLOWER
I20260925 15:22:47.621730 16180 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9426d03e35647be9b6fea84c8c8ee75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9426d03e35647be9b6fea84c8c8ee75" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 34085 } }
I20260925 15:22:47.621824 16180 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9426d03e35647be9b6fea84c8c8ee75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:22:47.621884 16180 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9426d03e35647be9b6fea84c8c8ee75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:22:47.621963 16180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9426d03e35647be9b6fea84c8c8ee75 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:22:47.622720 16180 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9426d03e35647be9b6fea84c8c8ee75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9426d03e35647be9b6fea84c8c8ee75" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 34085 } }
I20260925 15:22:47.622852 16180 leader_election.cc:302] T 00000000000000000000000000000000 P b9426d03e35647be9b6fea84c8c8ee75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9426d03e35647be9b6fea84c8c8ee75; no voters:
I20260925 15:22:47.623078 16180 leader_election.cc:288] T 00000000000000000000000000000000 P b9426d03e35647be9b6fea84c8c8ee75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:22:47.623111 16185 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9426d03e35647be9b6fea84c8c8ee75 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:22:47.623245 16185 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9426d03e35647be9b6fea84c8c8ee75 [term 1 LEADER]: Becoming Leader. State: Replica: b9426d03e35647be9b6fea84c8c8ee75, State: Running, Role: LEADER
I20260925 15:22:47.623405 16180 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9426d03e35647be9b6fea84c8c8ee75 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:22:47.623392 16185 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9426d03e35647be9b6fea84c8c8ee75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9426d03e35647be9b6fea84c8c8ee75" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 34085 } }
I20260925 15:22:47.623937 16186 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9426d03e35647be9b6fea84c8c8ee75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9426d03e35647be9b6fea84c8c8ee75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9426d03e35647be9b6fea84c8c8ee75" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 34085 } } }
I20260925 15:22:47.624152 16187 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9426d03e35647be9b6fea84c8c8ee75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9426d03e35647be9b6fea84c8c8ee75. Latest consensus state: current_term: 1 leader_uuid: "b9426d03e35647be9b6fea84c8c8ee75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9426d03e35647be9b6fea84c8c8ee75" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 34085 } } }
I20260925 15:22:47.624236 16186 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9426d03e35647be9b6fea84c8c8ee75 [sys.catalog]: This master's current role is: LEADER
I20260925 15:22:47.624361 16187 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9426d03e35647be9b6fea84c8c8ee75 [sys.catalog]: This master's current role is: LEADER
I20260925 15:22:47.624548 16194 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:22:47.625128 16194 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:22:47.625449 16180 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Sep 25 15:22:47 dist-test-slave-4bk3 krb5kdc[15668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349767, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:22:47.632946 16194 catalog_manager.cc:1347] Generated new cluster ID: 49dea86c367f41259d8224366acb43a3
I20260925 15:22:47.633107 16194 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:22:47.633850 16184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:47.620467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52614 (local address 127.3.211.254:34085)
0925 15:22:47.620647 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:47.620656 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:47.620683 (+ 27us) server_negotiation.cc:407] Connection header received
0925 15:22:47.621326 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:47.621342 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:47.621401 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:47.621589 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:47.622513 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:47.623752 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:47.624895 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:47.625509 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:47.625621 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:47.628926 (+ 3305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:47.628976 (+ 50us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:47.628976 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:47.629068 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:47.631572 (+ 2504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:47.632105 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:47.632109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:47.632109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:47.632228 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:47.632523 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:47.632524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:47.632524 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:47.632894 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:47.633060 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:47.633416 (+ 356us) server_negotiation.cc:299] Negotiation successful
0925 15:22:47.633590 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":88,"threads_started":1}
I20260925 15:22:47.652613 16194 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:22:47.653450 16194 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:22:47.659798 16194 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9426d03e35647be9b6fea84c8c8ee75: Generated new TSK 0
I20260925 15:22:47.660048 16194 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.211.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab.
Entry for principal kudu/127.3.211.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab.
I20260925 15:22:48.388986 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.211.212:57279/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.193:0
--local_ip_for_outbound_sockets=127.3.211.193
--webserver_interface=127.3.211.193
--webserver_port=0
--tserver_master_addrs=127.3.211.254:34085
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab
--principal=kudu/127.3.211.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:41295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:22:48.512439 16225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:22:48.512701 16225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:22:48.512729 16225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:22:48.514262 16225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:22:48.514333 16225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:22:48.514365 16225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.193
I20260925 15:22:48.516464 16225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:41295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.211.212:57279/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab
--principal=kudu/127.3.211.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.211.193
--webserver_port=0
--tserver_master_addrs=127.3.211.254:34085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.193
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:22:48.516752 16225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:22:48.517086 16225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:22:48.522622 16230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:48.522657 16233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:48.522787 16231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:22:48.523574 16225 server_base.cc:1034] running on GCE node
I20260925 15:22:48.523941 16225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:22:48.524312 16225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:22:48.525521 16225 hybrid_clock.cc:648] HybridClock initialized: now 1790349768525483 us; error 39 us; skew 500 ppm
Sep 25 15:22:48 dist-test-slave-4bk3 krb5kdc[15668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349768, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:22:48.529305 16225 init.cc:377] Logged in from keytab as kudu/127.3.211.193@KRBTEST.COM (short username kudu)
I20260925 15:22:48.529980 16225 webserver.cc:460] Webserver started at http://127.3.211.193:40801/ using document root <none> and password file <none>
I20260925 15:22:48.530288 16225 fs_manager.cc:359] Metadata directory not provided
I20260925 15:22:48.530386 16225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:22:48.530539 16225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:22:48 dist-test-slave-4bk3 krb5kdc[15668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349768, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.193@KRBTEST.COM for HTTP/127.3.211.212@KRBTEST.COM
I20260925 15:22:48.547412 16225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-0/data/instance:
uuid: "2184196ed67a4d1fbbdfa9a57d23efa7"
format_stamp: "Formatted at 2026-09-25 15:22:48 on dist-test-slave-4bk3"
server_key: "3827f1e0c3c14e3a77338f869953bff7"
server_key_iv: "ab899f6dc15a378183a69f8375decb94"
server_key_version: "kuduclusterkey@0"
I20260925 15:22:48.547955 16225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-0/wal/instance:
uuid: "2184196ed67a4d1fbbdfa9a57d23efa7"
format_stamp: "Formatted at 2026-09-25 15:22:48 on dist-test-slave-4bk3"
server_key: "3827f1e0c3c14e3a77338f869953bff7"
server_key_iv: "ab899f6dc15a378183a69f8375decb94"
server_key_version: "kuduclusterkey@0"
I20260925 15:22:48.550249 16225 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:22:48.564697 16240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:48.565274 16225 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 15:22:48.565443 16225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-0/wal
uuid: "2184196ed67a4d1fbbdfa9a57d23efa7"
format_stamp: "Formatted at 2026-09-25 15:22:48 on dist-test-slave-4bk3"
server_key: "3827f1e0c3c14e3a77338f869953bff7"
server_key_iv: "ab899f6dc15a378183a69f8375decb94"
server_key_version: "kuduclusterkey@0"
I20260925 15:22:48.565564 16225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:22:48.589692 16225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:22:48.592463 16225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:22:48.592723 16225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:22:48.593073 16225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:22:48.593521 16225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:22:48.593617 16225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:48.593729 16225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:22:48.593842 16225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:48.602061 16225 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.193:33757
I20260925 15:22:48.602191 16355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.193:33757 every 8 connection(s)
I20260925 15:22:48.602581 16225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:22:48.604908 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 16225
I20260925 15:22:48.605028 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-0/wal/instance
I20260925 15:22:48.605187 3919 external_mini_cluster.cc:1428] Setting key 120ddbcae9eb64105d19a5acb37995dd
Sep 25 15:22:48 dist-test-slave-4bk3 krb5kdc[15668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349768, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.193@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:22:48.616063 16358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:48.603666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34085 (local address 127.3.211.193:33665)
0925 15:22:48.603932 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:48.603988 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:48.604724 (+ 736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:48.605220 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:48.605225 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:48.605492 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:22:48.605793 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:48.605799 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:48.606815 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:48.606816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:48.608151 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:48.608155 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:48.608759 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:48.608760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:48.608921 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:48.609609 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:22:48.609622 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:22:48.611572 (+ 1950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:22:48.614028 (+ 2456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:48.614040 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:48.614042 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:48.614418 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:48.614690 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:48.614691 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:48.614691 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:48.614830 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:48.615305 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:22:48.615313 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:22:48.615510 (+ 197us) client_negotiation.cc:769] Sending connection context
0925 15:22:48.615611 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 15:22:48.615802 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":65,"threads_started":1}
I20260925 15:22:48.617012 16359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:48.603699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:33665 (local address 127.3.211.254:34085)
0925 15:22:48.603853 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:48.603856 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:48.604113 (+ 257us) server_negotiation.cc:407] Connection header received
0925 15:22:48.604895 (+ 782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:48.604897 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:48.604914 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:48.605015 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:48.605884 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:48.606690 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:48.608255 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:48.608637 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:48.608718 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:48.611727 (+ 3009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:48.611730 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:48.611730 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:48.611740 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:48.613889 (+ 2149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:48.614526 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:48.614527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:48.614528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:48.614615 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:48.614914 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:48.614914 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:48.614915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:48.615198 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:48.616738 (+ 1540us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:48.616790 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 15:22:48.616899 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":89,"threads_started":1}
I20260925 15:22:48.617619 16356 heartbeater.cc:347] Connected to a master server at 127.3.211.254:34085
I20260925 15:22:48.617784 16356 heartbeater.cc:464] Registering TS with master...
I20260925 15:22:48.618057 16356 heartbeater.cc:511] Master 127.3.211.254:34085 requested a full tablet report, sending...
I20260925 15:22:48.618710 16144 ts_manager.cc:194] Registered new tserver with Master: 2184196ed67a4d1fbbdfa9a57d23efa7 (127.3.211.193:33757)
WARNING: no policy specified for kudu/127.3.211.194@KRBTEST.COM; defaulting to no policy
I20260925 15:22:48.619666 16144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.193@KRBTEST.COM'} at 127.3.211.193:33665
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab.
Entry for principal kudu/127.3.211.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab.
I20260925 15:22:48.644572 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.211.212:57279/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.194:0
--local_ip_for_outbound_sockets=127.3.211.194
--webserver_interface=127.3.211.194
--webserver_port=0
--tserver_master_addrs=127.3.211.254:34085
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab
--principal=kudu/127.3.211.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:41295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:22:48.766821 16362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:22:48.767138 16362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:22:48.767177 16362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:22:48.769161 16362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:22:48.769277 16362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:22:48.769345 16362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.194
I20260925 15:22:48.771906 16362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:41295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.211.212:57279/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab
--principal=kudu/127.3.211.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.211.194
--webserver_port=0
--tserver_master_addrs=127.3.211.254:34085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.194
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:22:48.772449 16362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:22:48.772785 16362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Sep 25 15:22:48 dist-test-slave-4bk3 krb5kdc[15668](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790349768, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 15:22:48.779732 16370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:48.781555 16369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:48.781767 16372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:22:48.782171 16362 server_base.cc:1034] running on GCE node
I20260925 15:22:48.782399 16362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:22:48.782696 16362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:22:48.786001 16362 hybrid_clock.cc:648] HybridClock initialized: now 1790349768785977 us; error 46 us; skew 500 ppm
Sep 25 15:22:48 dist-test-slave-4bk3 krb5kdc[15668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349768, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:22:48.789789 16362 init.cc:377] Logged in from keytab as kudu/127.3.211.194@KRBTEST.COM (short username kudu)
I20260925 15:22:48.790362 16362 webserver.cc:460] Webserver started at http://127.3.211.194:46065/ using document root <none> and password file <none>
I20260925 15:22:48.790625 16362 fs_manager.cc:359] Metadata directory not provided
I20260925 15:22:48.790874 16362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:22:48.791083 16362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:22:48 dist-test-slave-4bk3 krb5kdc[15668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349768, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.194@KRBTEST.COM for HTTP/127.3.211.212@KRBTEST.COM
I20260925 15:22:48.807513 16362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-1/data/instance:
uuid: "0adb59e3f7b242c29e58a71dc004b245"
format_stamp: "Formatted at 2026-09-25 15:22:48 on dist-test-slave-4bk3"
server_key: "65671178ff22a7bf3f4eead54571a7f4"
server_key_iv: "6f1d3d656854de6e1f24bc3564b00a8f"
server_key_version: "kuduclusterkey@0"
I20260925 15:22:48.808225 16362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-1/wal/instance:
uuid: "0adb59e3f7b242c29e58a71dc004b245"
format_stamp: "Formatted at 2026-09-25 15:22:48 on dist-test-slave-4bk3"
server_key: "65671178ff22a7bf3f4eead54571a7f4"
server_key_iv: "6f1d3d656854de6e1f24bc3564b00a8f"
server_key_version: "kuduclusterkey@0"
I20260925 15:22:48.810278 16362 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260925 15:22:48.828774 16379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:48.829087 16362 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 15:22:48.829257 16362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-1/wal
uuid: "0adb59e3f7b242c29e58a71dc004b245"
format_stamp: "Formatted at 2026-09-25 15:22:48 on dist-test-slave-4bk3"
server_key: "65671178ff22a7bf3f4eead54571a7f4"
server_key_iv: "6f1d3d656854de6e1f24bc3564b00a8f"
server_key_version: "kuduclusterkey@0"
I20260925 15:22:48.829396 16362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:22:48.873536 16362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:22:48.876199 16362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:22:48.876446 16362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:22:48.876775 16362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:22:48.877206 16362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:22:48.877305 16362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:48.877390 16362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:22:48.877450 16362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:48.887598 16362 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.194:36321
I20260925 15:22:48.888212 16362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
I20260925 15:22:48.889863 16493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.194:36321 every 8 connection(s)
I20260925 15:22:48.894498 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 16362
I20260925 15:22:48.894601 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-1/wal/instance
I20260925 15:22:48.894804 3919 external_mini_cluster.cc:1428] Setting key 4f4d3b52d5088d951564c0ff6f5b8dde
Sep 25 15:22:48 dist-test-slave-4bk3 krb5kdc[15668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349768, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.194@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:22:48.902055 16359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:48.889763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:58049 (local address 127.3.211.254:34085)
0925 15:22:48.889831 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:48.889833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:48.890224 (+ 391us) server_negotiation.cc:407] Connection header received
0925 15:22:48.890956 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:48.890957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:48.890973 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:48.891011 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:48.891904 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:48.892738 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:48.894254 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:48.894564 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:48.894617 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:48.898301 (+ 3684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:48.898303 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:48.898304 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:48.898313 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:48.900161 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:48.900801 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:48.900802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:48.900803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:48.900884 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:48.901200 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:48.901202 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:48.901202 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:48.901357 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:48.901405 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:48.901823 (+ 418us) server_negotiation.cc:299] Negotiation successful
0925 15:22:48.901947 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 15:22:48.902838 16496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:48.889470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34085 (local address 127.3.211.194:58049)
0925 15:22:48.890089 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:48.890134 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:48.890809 (+ 675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:48.891235 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:48.891240 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:48.891498 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:22:48.891780 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:48.891785 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:48.892859 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:48.892860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:48.894150 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:48.894153 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:48.894822 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:48.894822 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:48.895128 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:48.895763 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:22:48.895775 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:22:48.898140 (+ 2365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:22:48.900292 (+ 2152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:48.900304 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:48.900307 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:48.900698 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:48.900961 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:48.900962 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:48.900962 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:48.901117 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:48.901459 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:22:48.901466 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:22:48.901652 (+ 186us) client_negotiation.cc:769] Sending connection context
0925 15:22:48.902276 (+ 624us) client_negotiation.cc:240] Negotiation successful
0925 15:22:48.902539 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":93,"threads_started":1}
I20260925 15:22:48.903425 16494 heartbeater.cc:347] Connected to a master server at 127.3.211.254:34085
I20260925 15:22:48.903551 16494 heartbeater.cc:464] Registering TS with master...
I20260925 15:22:48.903754 16494 heartbeater.cc:511] Master 127.3.211.254:34085 requested a full tablet report, sending...
I20260925 15:22:48.904246 16144 ts_manager.cc:194] Registered new tserver with Master: 0adb59e3f7b242c29e58a71dc004b245 (127.3.211.194:36321)
I20260925 15:22:48.904676 16144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.194@KRBTEST.COM'} at 127.3.211.194:58049
WARNING: no policy specified for kudu/127.3.211.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab.
Entry for principal kudu/127.3.211.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab.
I20260925 15:22:48.929533 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.211.212:57279/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.195:0
--local_ip_for_outbound_sockets=127.3.211.195
--webserver_interface=127.3.211.195
--webserver_port=0
--tserver_master_addrs=127.3.211.254:34085
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab
--principal=kudu/127.3.211.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:41295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:22:49.025763 16499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:22:49.026048 16499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:22:49.026232 16499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:22:49.027667 16499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:22:49.027765 16499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:22:49.027815 16499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.195
I20260925 15:22:49.029803 16499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:41295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.211.212:57279/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab
--principal=kudu/127.3.211.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.211.195
--webserver_port=0
--tserver_master_addrs=127.3.211.254:34085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.195
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:22:49.030284 16499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:22:49.030629 16499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:22:49.038146 16505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:49.038538 16504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:49.039235 16507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:22:49.039893 16499 server_base.cc:1034] running on GCE node
I20260925 15:22:49.040231 16499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:22:49.040585 16499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:22:49.041775 16499 hybrid_clock.cc:648] HybridClock initialized: now 1790349769041753 us; error 36 us; skew 500 ppm
Sep 25 15:22:49 dist-test-slave-4bk3 krb5kdc[15668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349769, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:22:49.045437 16499 init.cc:377] Logged in from keytab as kudu/127.3.211.195@KRBTEST.COM (short username kudu)
I20260925 15:22:49.046043 16499 webserver.cc:460] Webserver started at http://127.3.211.195:37359/ using document root <none> and password file <none>
I20260925 15:22:49.046303 16499 fs_manager.cc:359] Metadata directory not provided
I20260925 15:22:49.046360 16499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:22:49.046492 16499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:22:49 dist-test-slave-4bk3 krb5kdc[15668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349769, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.195@KRBTEST.COM for HTTP/127.3.211.212@KRBTEST.COM
I20260925 15:22:49.060760 16499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-2/data/instance:
uuid: "eed3118981c64bb891264464ec90f208"
format_stamp: "Formatted at 2026-09-25 15:22:49 on dist-test-slave-4bk3"
server_key: "7b90764c78e9473eb34a6934d8376c46"
server_key_iv: "db796bad8e544a9816d3edb033df063f"
server_key_version: "kuduclusterkey@0"
I20260925 15:22:49.061246 16499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-2/wal/instance:
uuid: "eed3118981c64bb891264464ec90f208"
format_stamp: "Formatted at 2026-09-25 15:22:49 on dist-test-slave-4bk3"
server_key: "7b90764c78e9473eb34a6934d8376c46"
server_key_iv: "db796bad8e544a9816d3edb033df063f"
server_key_version: "kuduclusterkey@0"
I20260925 15:22:49.070372 16499 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.002s sys 0.000s
I20260925 15:22:49.084823 16514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:49.087095 16499 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260925 15:22:49.087230 16499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-2/wal
uuid: "eed3118981c64bb891264464ec90f208"
format_stamp: "Formatted at 2026-09-25 15:22:49 on dist-test-slave-4bk3"
server_key: "7b90764c78e9473eb34a6934d8376c46"
server_key_iv: "db796bad8e544a9816d3edb033df063f"
server_key_version: "kuduclusterkey@0"
I20260925 15:22:49.087328 16499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:22:49.102428 16499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:22:49.104799 16499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:22:49.104986 16499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:22:49.105281 16499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:22:49.105693 16499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:22:49.105749 16499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:49.105818 16499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:22:49.105847 16499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:49.113072 16499 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.195:34233
I20260925 15:22:49.113641 16499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
I20260925 15:22:49.115325 16627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.195:34233 every 8 connection(s)
I20260925 15:22:49.117198 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 16499
I20260925 15:22:49.117304 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-2/wal/instance
I20260925 15:22:49.117506 3919 external_mini_cluster.cc:1428] Setting key 51ba5c6652c36d149960431ef21d466c
Sep 25 15:22:49 dist-test-slave-4bk3 krb5kdc[15668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349769, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.195@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:22:49.132009 16359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:49.120448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:36661 (local address 127.3.211.254:34085)
0925 15:22:49.120503 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:49.120505 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:49.120724 (+ 219us) server_negotiation.cc:407] Connection header received
0925 15:22:49.121454 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:49.121455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:49.121469 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:49.121500 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:49.122414 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:49.123238 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:49.125050 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:49.125355 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:49.125442 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:49.128307 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:49.128310 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:49.128310 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:49.128319 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:49.130174 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:49.130763 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:49.130764 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:49.130764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:49.130828 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:49.131172 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:49.131173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:49.131173 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:49.131324 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:49.131369 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:49.131794 (+ 425us) server_negotiation.cc:299] Negotiation successful
0925 15:22:49.131904 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 15:22:49.132994 16630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:49.116045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34085 (local address 127.3.211.195:36661)
0925 15:22:49.120557 (+ 4512us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:49.120625 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:49.121322 (+ 697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:49.121710 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:49.121714 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:49.121990 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:22:49.122299 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:49.122305 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:49.123381 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:49.123382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:49.124940 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:49.124944 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:49.125491 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:49.125492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:49.125646 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:49.126324 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:22:49.126335 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:22:49.128161 (+ 1826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:22:49.130292 (+ 2131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:49.130303 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:49.130306 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:49.130662 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:49.130903 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:49.130904 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:49.130904 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:49.131089 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:49.131426 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:22:49.131434 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:22:49.131620 (+ 186us) client_negotiation.cc:769] Sending connection context
0925 15:22:49.132526 (+ 906us) client_negotiation.cc:240] Negotiation successful
0925 15:22:49.132730 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4402,"thread_start_us":92,"threads_started":1}
I20260925 15:22:49.133834 16628 heartbeater.cc:347] Connected to a master server at 127.3.211.254:34085
I20260925 15:22:49.133967 16628 heartbeater.cc:464] Registering TS with master...
I20260925 15:22:49.134197 16628 heartbeater.cc:511] Master 127.3.211.254:34085 requested a full tablet report, sending...
I20260925 15:22:49.134625 16144 ts_manager.cc:194] Registered new tserver with Master: eed3118981c64bb891264464ec90f208 (127.3.211.195:34233)
I20260925 15:22:49.135195 16144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.195@KRBTEST.COM'} at 127.3.211.195:36661
I20260925 15:22:49.140991 3919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:22:49.141702 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 16225
I20260925 15:22:49.150324 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 16362
I20260925 15:22:49.157829 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 16499
I20260925 15:22:49.162680 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 16112
I20260925 15:22:49.170308 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:34085
--webserver_interface=127.3.211.254
--webserver_port=43199
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:41295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.211.254:34085
--ranger_config_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.211.212:57279/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.211.212:57279/kms with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:22:49.258458 16631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:22:49.258714 16631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:22:49.258738 16631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:22:49.259960 16631 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:22:49.260015 16631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:22:49.260030 16631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:22:49.260043 16631 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:22:49.260056 16631 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:22:49.261758 16631 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:41295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.211.212:57279/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.211.254:34085
--ranger_config_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:34085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.211.254
--webserver_port=43199
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:22:49.261981 16631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:22:49.262167 16631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:22:49.268034 16636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:49.268013 16639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:49.267997 16637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:22:49.268476 16631 server_base.cc:1034] running on GCE node
I20260925 15:22:49.268774 16631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:22:49.269076 16631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:22:49.270236 16631 hybrid_clock.cc:648] HybridClock initialized: now 1790349769270226 us; error 36 us; skew 500 ppm
Sep 25 15:22:49 dist-test-slave-4bk3 krb5kdc[15668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349769, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:22:49.273855 16631 init.cc:377] Logged in from keytab as kudu/127.3.211.254@KRBTEST.COM (short username kudu)
I20260925 15:22:49.274482 16631 webserver.cc:460] Webserver started at http://127.3.211.254:43199/ using document root <none> and password file <none>
I20260925 15:22:49.274745 16631 fs_manager.cc:359] Metadata directory not provided
I20260925 15:22:49.274828 16631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 15:22:49 dist-test-slave-4bk3 krb5kdc[15668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349769, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.254@KRBTEST.COM for HTTP/127.3.211.212@KRBTEST.COM
I20260925 15:22:49.320361 16631 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:22:49.321339 16647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:49.321534 16631 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:22:49.321633 16631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/master-0/wal
uuid: "b9426d03e35647be9b6fea84c8c8ee75"
format_stamp: "Formatted at 2026-09-25 15:22:47 on dist-test-slave-4bk3"
server_key: "d08a053a083ce6149eb83f4686ad37a2"
server_key_iv: "16aab3315534929d909e7dbcaaa506a5"
server_key_version: "kuduclusterkey@0"
I20260925 15:22:49.322000 16631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:22:49.341951 16631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:22:49.344192 16631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:22:49.344398 16631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:22:49.349308 16631 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.254:34085
I20260925 15:22:49.349347 16699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.254:34085 every 8 connection(s)
I20260925 15:22:49.349736 16631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
I20260925 15:22:49.350922 16700 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:22:49.351611 16700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9426d03e35647be9b6fea84c8c8ee75: Bootstrap starting.
I20260925 15:22:49.356918 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 16631
I20260925 15:22:49.357301 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.211.212:57279/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.193:33757
--local_ip_for_outbound_sockets=127.3.211.193
--tserver_master_addrs=127.3.211.254:34085
--webserver_port=40801
--webserver_interface=127.3.211.193
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab
--principal=kudu/127.3.211.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:41295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.211.212:57279/kms with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:22:49.359926 16700 log.cc:824] T 00000000000000000000000000000000 P b9426d03e35647be9b6fea84c8c8ee75: Log is configured to *not* fsync() on all Append() calls
I20260925 15:22:49.361956 16700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9426d03e35647be9b6fea84c8c8ee75: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:22:49.362290 16700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9426d03e35647be9b6fea84c8c8ee75: Bootstrap complete.
I20260925 15:22:49.365046 16700 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9426d03e35647be9b6fea84c8c8ee75 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9426d03e35647be9b6fea84c8c8ee75" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 34085 } }
I20260925 15:22:49.365343 16700 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9426d03e35647be9b6fea84c8c8ee75 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9426d03e35647be9b6fea84c8c8ee75, State: Initialized, Role: FOLLOWER
I20260925 15:22:49.365546 16700 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9426d03e35647be9b6fea84c8c8ee75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9426d03e35647be9b6fea84c8c8ee75" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 34085 } }
I20260925 15:22:49.365645 16700 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9426d03e35647be9b6fea84c8c8ee75 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:22:49.365700 16700 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9426d03e35647be9b6fea84c8c8ee75 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:22:49.365748 16700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9426d03e35647be9b6fea84c8c8ee75 [term 1 FOLLOWER]: Advancing to term 2
I20260925 15:22:49.366849 16700 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9426d03e35647be9b6fea84c8c8ee75 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9426d03e35647be9b6fea84c8c8ee75" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 34085 } }
I20260925 15:22:49.366986 16700 leader_election.cc:302] T 00000000000000000000000000000000 P b9426d03e35647be9b6fea84c8c8ee75 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9426d03e35647be9b6fea84c8c8ee75; no voters:
I20260925 15:22:49.367220 16700 leader_election.cc:288] T 00000000000000000000000000000000 P b9426d03e35647be9b6fea84c8c8ee75 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 15:22:49.367262 16704 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9426d03e35647be9b6fea84c8c8ee75 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 15:22:49.367374 16704 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9426d03e35647be9b6fea84c8c8ee75 [term 2 LEADER]: Becoming Leader. State: Replica: b9426d03e35647be9b6fea84c8c8ee75, State: Running, Role: LEADER
I20260925 15:22:49.367512 16700 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9426d03e35647be9b6fea84c8c8ee75 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:22:49.367502 16704 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9426d03e35647be9b6fea84c8c8ee75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9426d03e35647be9b6fea84c8c8ee75" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 34085 } }
I20260925 15:22:49.367926 16705 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9426d03e35647be9b6fea84c8c8ee75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b9426d03e35647be9b6fea84c8c8ee75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9426d03e35647be9b6fea84c8c8ee75" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 34085 } } }
I20260925 15:22:49.368036 16705 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9426d03e35647be9b6fea84c8c8ee75 [sys.catalog]: This master's current role is: LEADER
I20260925 15:22:49.367935 16706 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9426d03e35647be9b6fea84c8c8ee75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9426d03e35647be9b6fea84c8c8ee75. Latest consensus state: current_term: 2 leader_uuid: "b9426d03e35647be9b6fea84c8c8ee75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9426d03e35647be9b6fea84c8c8ee75" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 34085 } } }
I20260925 15:22:49.368112 16706 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9426d03e35647be9b6fea84c8c8ee75 [sys.catalog]: This master's current role is: LEADER
I20260925 15:22:49.368577 16709 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:22:49.369117 16709 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:22:49.369279 16709 catalog_manager.cc:1259] Loaded cluster ID: 49dea86c367f41259d8224366acb43a3
I20260925 15:22:49.369369 16709 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:22:49.369620 16700 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 15:22:49.373595 16709 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:22:49.374626 16709 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b9426d03e35647be9b6fea84c8c8ee75: Loaded TSK: 0
I20260925 15:22:49.375165 16709 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 15:22:49.479954 16702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:22:49.480265 16702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:22:49.480312 16702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:22:49.482285 16702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:22:49.482367 16702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:22:49.482415 16702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.193
I20260925 15:22:49.485855 16702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:41295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.211.212:57279/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab
--principal=kudu/127.3.211.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.193:33757
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.211.193
--webserver_port=40801
--tserver_master_addrs=127.3.211.254:34085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.193
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:22:49.486634 16702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:22:49.487087 16702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:22:49.494211 16737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:49.494211 16738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:49.494211 16740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:22:49.495785 16702 server_base.cc:1034] running on GCE node
I20260925 15:22:49.496071 16702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:22:49.496443 16702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:22:49.497648 16702 hybrid_clock.cc:648] HybridClock initialized: now 1790349769497617 us; error 48 us; skew 500 ppm
Sep 25 15:22:49 dist-test-slave-4bk3 krb5kdc[15668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349769, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:22:49.501307 16702 init.cc:377] Logged in from keytab as kudu/127.3.211.193@KRBTEST.COM (short username kudu)
I20260925 15:22:49.501992 16702 webserver.cc:460] Webserver started at http://127.3.211.193:40801/ using document root <none> and password file <none>
I20260925 15:22:49.502300 16702 fs_manager.cc:359] Metadata directory not provided
I20260925 15:22:49.502385 16702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 15:22:49 dist-test-slave-4bk3 krb5kdc[15668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349769, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.193@KRBTEST.COM for HTTP/127.3.211.212@KRBTEST.COM
I20260925 15:22:49.524868 16702 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.001s sys 0.000s
I20260925 15:22:49.525827 16748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:49.526036 16702 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 15:22:49.526127 16702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-0/wal
uuid: "2184196ed67a4d1fbbdfa9a57d23efa7"
format_stamp: "Formatted at 2026-09-25 15:22:48 on dist-test-slave-4bk3"
server_key: "3827f1e0c3c14e3a77338f869953bff7"
server_key_iv: "ab899f6dc15a378183a69f8375decb94"
server_key_version: "kuduclusterkey@0"
I20260925 15:22:49.527920 16702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:22:49.545243 16702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:22:49.548280 16702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:22:49.548635 16702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:22:49.548950 16702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:22:49.549592 16702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:22:49.549927 16702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:49.550012 16702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:22:49.550034 16702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:49.558537 16702 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.193:33757
I20260925 15:22:49.558638 16861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.193:33757 every 8 connection(s)
I20260925 15:22:49.559139 16702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:22:49.566165 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 16702
I20260925 15:22:49.566493 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.211.212:57279/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.194:36321
--local_ip_for_outbound_sockets=127.3.211.194
--tserver_master_addrs=127.3.211.254:34085
--webserver_port=46065
--webserver_interface=127.3.211.194
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab
--principal=kudu/127.3.211.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:41295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.211.212:57279/kms with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:22:49 dist-test-slave-4bk3 krb5kdc[15668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349769, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.193@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:22:49.583051 16865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:49.560915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34085 (local address 127.3.211.193:57435)
0925 15:22:49.561758 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:49.561816 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:49.562571 (+ 755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:49.565183 (+ 2612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:49.565188 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:49.565475 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:22:49.565769 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:49.565776 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:49.570538 (+ 4762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:49.570540 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:49.571935 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:49.571942 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:49.572953 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:49.572955 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:49.573127 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:49.573795 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:22:49.573806 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:22:49.575602 (+ 1796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:22:49.580915 (+ 5313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:49.580928 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:49.580931 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:49.581295 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:49.581628 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:49.581629 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:49.581630 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:49.581790 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:49.582316 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:22:49.582322 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:22:49.582484 (+ 162us) client_negotiation.cc:769] Sending connection context
0925 15:22:49.582594 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 15:22:49.582782 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":789,"thread_start_us":396,"threads_started":1}
I20260925 15:22:49.584259 16864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:49.560955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:57435 (local address 127.3.211.254:34085)
0925 15:22:49.564598 (+ 3643us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:49.564610 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:49.564664 (+ 54us) server_negotiation.cc:407] Connection header received
0925 15:22:49.564836 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:49.564845 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:49.564911 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:49.565110 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:49.568588 (+ 3478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:49.570342 (+ 1754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:49.572113 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:49.572797 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:49.573471 (+ 674us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:49.575753 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:49.575759 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:49.575759 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:49.575773 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:49.580791 (+ 5018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:49.581408 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:49.581412 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:49.581412 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:49.581541 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:49.581874 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:49.581875 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:49.581875 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:49.582222 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:49.583714 (+ 1492us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:49.583805 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 15:22:49.583943 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3566,"thread_start_us":349,"threads_started":1}
I20260925 15:22:49.588645 16862 heartbeater.cc:347] Connected to a master server at 127.3.211.254:34085
I20260925 15:22:49.588783 16862 heartbeater.cc:464] Registering TS with master...
I20260925 15:22:49.589025 16862 heartbeater.cc:511] Master 127.3.211.254:34085 requested a full tablet report, sending...
W20260925 15:22:49.590128 16862 heartbeater.cc:644] Failed to heartbeat to 127.3.211.254:34085 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 15:22:49.590209 16862 heartbeater.cc:464] Registering TS with master...
I20260925 15:22:49.590284 16862 heartbeater.cc:511] Master 127.3.211.254:34085 requested a full tablet report, sending...
I20260925 15:22:49.590880 16862 heartbeater.cc:464] Registering TS with master...
I20260925 15:22:49.590976 16862 heartbeater.cc:511] Master 127.3.211.254:34085 requested a full tablet report, sending...
W20260925 15:22:49.591424 16862 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 15:22:49.691056 16866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:22:49.691658 16866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:22:49.691720 16866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:22:49.693687 16866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:22:49.693763 16866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:22:49.693809 16866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.194
I20260925 15:22:49.696249 16866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:41295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.211.212:57279/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab
--principal=kudu/127.3.211.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.194:36321
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.211.194
--webserver_port=46065
--tserver_master_addrs=127.3.211.254:34085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.194
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:22:49.696748 16866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:22:49.697032 16866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:22:49.702558 16872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:49.702560 16874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:49.702558 16871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:22:49.703544 16866 server_base.cc:1034] running on GCE node
I20260925 15:22:49.703804 16866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:22:49.704097 16866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:22:49.705281 16866 hybrid_clock.cc:648] HybridClock initialized: now 1790349769705242 us; error 55 us; skew 500 ppm
Sep 25 15:22:49 dist-test-slave-4bk3 krb5kdc[15668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349769, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:22:49.708216 16866 init.cc:377] Logged in from keytab as kudu/127.3.211.194@KRBTEST.COM (short username kudu)
I20260925 15:22:49.708789 16866 webserver.cc:460] Webserver started at http://127.3.211.194:46065/ using document root <none> and password file <none>
I20260925 15:22:49.709070 16866 fs_manager.cc:359] Metadata directory not provided
I20260925 15:22:49.709162 16866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 15:22:49 dist-test-slave-4bk3 krb5kdc[15668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349769, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.194@KRBTEST.COM for HTTP/127.3.211.212@KRBTEST.COM
I20260925 15:22:49.732496 16866 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260925 15:22:49.734115 16882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:49.734325 16866 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:22:49.734424 16866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-1/wal
uuid: "0adb59e3f7b242c29e58a71dc004b245"
format_stamp: "Formatted at 2026-09-25 15:22:48 on dist-test-slave-4bk3"
server_key: "65671178ff22a7bf3f4eead54571a7f4"
server_key_iv: "6f1d3d656854de6e1f24bc3564b00a8f"
server_key_version: "kuduclusterkey@0"
I20260925 15:22:49.734848 16866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:22:49.755785 16866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:22:49.758531 16866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:22:49.758750 16866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:22:49.759054 16866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:22:49.759450 16866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:22:49.759508 16866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:49.759550 16866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:22:49.759578 16866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:49.767218 16866 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.194:36321
I20260925 15:22:49.767298 16995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.194:36321 every 8 connection(s)
I20260925 15:22:49.767808 16866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
I20260925 15:22:49.771785 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 16866
I20260925 15:22:49.772115 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.211.212:57279/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.195:34233
--local_ip_for_outbound_sockets=127.3.211.195
--tserver_master_addrs=127.3.211.254:34085
--webserver_port=37359
--webserver_interface=127.3.211.195
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab
--principal=kudu/127.3.211.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:41295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.211.212:57279/kms with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:22:49 dist-test-slave-4bk3 krb5kdc[15668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349769, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.194@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:22:49.780071 16864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:49.768681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:53583 (local address 127.3.211.254:34085)
0925 15:22:49.768752 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:49.768755 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:49.769091 (+ 336us) server_negotiation.cc:407] Connection header received
0925 15:22:49.769850 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:49.769853 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:49.769880 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:49.770002 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:49.770858 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:49.771665 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:49.773107 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:49.773341 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:49.773460 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:49.776558 (+ 3098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:49.776560 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:49.776561 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:49.776571 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:49.778425 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:49.778933 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:49.778934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:49.778935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:49.778998 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:49.779340 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:49.779341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:49.779342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:49.779475 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:49.779558 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:49.779874 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 15:22:49.779974 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 15:22:49.780283 16998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:49.768695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34085 (local address 127.3.211.194:53583)
0925 15:22:49.768909 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:49.768966 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:49.769671 (+ 705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:49.770130 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:49.770135 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:49.770392 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:22:49.770700 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:49.770706 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:49.771772 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:49.771774 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:49.772997 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:49.772999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:49.773491 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:49.773492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:49.773665 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:49.774381 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:22:49.774393 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:22:49.776219 (+ 1826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:22:49.778545 (+ 2326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:49.778558 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:49.778561 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:49.778845 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:49.779080 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:49.779081 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:49.779081 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:49.779276 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:49.779549 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:22:49.779557 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:22:49.779726 (+ 169us) client_negotiation.cc:769] Sending connection context
0925 15:22:49.779849 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 15:22:49.780018 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":106,"threads_started":1}
I20260925 15:22:49.780862 16996 heartbeater.cc:347] Connected to a master server at 127.3.211.254:34085
I20260925 15:22:49.781008 16996 heartbeater.cc:464] Registering TS with master...
I20260925 15:22:49.781247 16996 heartbeater.cc:511] Master 127.3.211.254:34085 requested a full tablet report, sending...
W20260925 15:22:49.781971 16996 heartbeater.cc:644] Failed to heartbeat to 127.3.211.254:34085 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 15:22:49.782076 16996 heartbeater.cc:464] Registering TS with master...
I20260925 15:22:49.782145 16996 heartbeater.cc:511] Master 127.3.211.254:34085 requested a full tablet report, sending...
I20260925 15:22:49.782776 16996 heartbeater.cc:464] Registering TS with master...
I20260925 15:22:49.782871 16996 heartbeater.cc:511] Master 127.3.211.254:34085 requested a full tablet report, sending...
W20260925 15:22:49.783223 16996 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 15:22:49.880815 16999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:22:49.881138 16999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:22:49.881181 16999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:22:49.882540 16999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:22:49.882630 16999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:22:49.882679 16999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.195
I20260925 15:22:49.884454 16999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:41295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.211.212:57279/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab
--principal=kudu/127.3.211.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.195:34233
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.211.195
--webserver_port=37359
--tserver_master_addrs=127.3.211.254:34085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.195
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:22:49.884742 16999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:22:49.885010 16999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:22:49.891147 17005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:49.893584 17006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:49.894354 17008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:22:49.900532 16999 server_base.cc:1034] running on GCE node
I20260925 15:22:49.900902 16999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:22:49.901295 16999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:22:49.902647 16999 hybrid_clock.cc:648] HybridClock initialized: now 1790349769902489 us; error 171 us; skew 500 ppm
Sep 25 15:22:49 dist-test-slave-4bk3 krb5kdc[15668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349769, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:22:49.908882 16999 init.cc:377] Logged in from keytab as kudu/127.3.211.195@KRBTEST.COM (short username kudu)
I20260925 15:22:49.909533 16999 webserver.cc:460] Webserver started at http://127.3.211.195:37359/ using document root <none> and password file <none>
I20260925 15:22:49.909819 16999 fs_manager.cc:359] Metadata directory not provided
I20260925 15:22:49.909917 16999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 15:22:49 dist-test-slave-4bk3 krb5kdc[15668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349769, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.195@KRBTEST.COM for HTTP/127.3.211.212@KRBTEST.COM
I20260925 15:22:49.930950 16999 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.001s
I20260925 15:22:49.938218 17016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:49.938494 16999 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.000s sys 0.000s
I20260925 15:22:49.938647 16999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-2/wal
uuid: "eed3118981c64bb891264464ec90f208"
format_stamp: "Formatted at 2026-09-25 15:22:49 on dist-test-slave-4bk3"
server_key: "7b90764c78e9473eb34a6934d8376c46"
server_key_iv: "db796bad8e544a9816d3edb033df063f"
server_key_version: "kuduclusterkey@0"
I20260925 15:22:49.939095 16999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:22:49.948652 16999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:22:49.951207 16999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:22:49.951400 16999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:22:49.951689 16999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:22:49.952080 16999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:22:49.952135 16999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:49.952179 16999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:22:49.952208 16999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:49.958829 16999 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.195:34233
I20260925 15:22:49.959340 16999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
I20260925 15:22:49.961686 17129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.195:34233 every 8 connection(s)
I20260925 15:22:49.962591 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 16999
Sep 25 15:22:49 dist-test-slave-4bk3 krb5kdc[15668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349769, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.195@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:22:49.975390 17133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:49.963044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52616 (local address 127.3.211.254:34085)
0925 15:22:49.964469 (+ 1425us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:49.964471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:49.964719 (+ 248us) server_negotiation.cc:407] Connection header received
0925 15:22:49.964820 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:49.964821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:49.964832 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:49.964877 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:49.965781 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:49.966610 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:49.967728 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:49.968024 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:49.968097 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:49.969769 (+ 1672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:49.969771 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:49.969771 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:49.969779 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:49.971402 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:49.974218 (+ 2816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:49.974219 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:49.974219 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:49.974294 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:49.974702 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:49.974703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:49.974704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:49.974869 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:49.975034 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:49.975171 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 15:22:49.975278 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1392,"thread_start_us":58,"threads_started":1}
I20260925 15:22:49.975631 17132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:49.961953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34085 (local address 127.3.211.195:33141)
0925 15:22:49.962166 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:49.962223 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:49.962929 (+ 706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:49.963380 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:49.963384 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:49.963646 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:22:49.963960 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:49.963967 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:49.965256 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:49.965257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:49.966465 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:49.966468 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:49.966935 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:49.966936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:49.967078 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:49.967716 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:22:49.967728 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:22:49.969746 (+ 2018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:22:49.973722 (+ 3976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:49.973738 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:49.973741 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:49.974106 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:49.974359 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:49.974360 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:49.974360 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:49.974482 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:49.974825 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:22:49.974831 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:22:49.975020 (+ 189us) client_negotiation.cc:769] Sending connection context
0925 15:22:49.975175 (+ 155us) client_negotiation.cc:240] Negotiation successful
0925 15:22:49.975358 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":90,"threads_started":1}
I20260925 15:22:49.975960 16864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:49.961965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:33141 (local address 127.3.211.254:34085)
0925 15:22:49.962028 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:49.962030 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:49.962323 (+ 293us) server_negotiation.cc:407] Connection header received
0925 15:22:49.963227 (+ 904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:49.963229 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:49.963242 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:49.963299 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:49.964098 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:49.965124 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:49.966595 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:49.966805 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:49.966898 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:49.969865 (+ 2967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:49.969867 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:49.969867 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:49.969875 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:49.973586 (+ 3711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:49.974196 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:49.974198 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:49.974198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:49.974278 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:49.974550 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:49.974551 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:49.974551 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:49.974723 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:49.974822 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:49.975786 (+ 964us) server_negotiation.cc:299] Negotiation successful
0925 15:22:49.975872 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 15:22:49.976420 17130 heartbeater.cc:347] Connected to a master server at 127.3.211.254:34085
I20260925 15:22:49.976522 17130 heartbeater.cc:464] Registering TS with master...
I20260925 15:22:49.976740 17130 heartbeater.cc:511] Master 127.3.211.254:34085 requested a full tablet report, sending...
W20260925 15:22:49.977609 17130 heartbeater.cc:644] Failed to heartbeat to 127.3.211.254:34085 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 15:22:49.977697 17130 heartbeater.cc:464] Registering TS with master...
I20260925 15:22:49.977772 17130 heartbeater.cc:511] Master 127.3.211.254:34085 requested a full tablet report, sending...
I20260925 15:22:49.978188 17130 heartbeater.cc:464] Registering TS with master...
I20260925 15:22:49.978288 17130 heartbeater.cc:511] Master 127.3.211.254:34085 requested a full tablet report, sending...
W20260925 15:22:49.978638 17130 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 15:22:50 dist-test-slave-4bk3 krb5kdc[15668](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790349770, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:22:50.591661 16862 heartbeater.cc:464] Registering TS with master...
I20260925 15:22:50.591845 16862 heartbeater.cc:511] Master 127.3.211.254:34085 requested a full tablet report, sending...
I20260925 15:22:50.592366 16663 ts_manager.cc:194] Registered new tserver with Master: 2184196ed67a4d1fbbdfa9a57d23efa7 (127.3.211.193:33757)
I20260925 15:22:50.593395 16663 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.193@KRBTEST.COM'} at 127.3.211.193:57435
I20260925 15:22:50.783463 16996 heartbeater.cc:464] Registering TS with master...
I20260925 15:22:50.783650 16996 heartbeater.cc:511] Master 127.3.211.254:34085 requested a full tablet report, sending...
I20260925 15:22:50.784057 16663 ts_manager.cc:194] Registered new tserver with Master: 0adb59e3f7b242c29e58a71dc004b245 (127.3.211.194:36321)
I20260925 15:22:50.784616 16663 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.194@KRBTEST.COM'} at 127.3.211.194:53583
Sep 25 15:22:50 dist-test-slave-4bk3 krb5kdc[15668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349770, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.254@KRBTEST.COM for HTTP/127.3.211.212@KRBTEST.COM
I20260925 15:22:50.978869 17130 heartbeater.cc:464] Registering TS with master...
I20260925 15:22:50.979032 17130 heartbeater.cc:511] Master 127.3.211.254:34085 requested a full tablet report, sending...
I20260925 15:22:50.979373 16663 ts_manager.cc:194] Registered new tserver with Master: eed3118981c64bb891264464ec90f208 (127.3.211.195:34233)
I20260925 15:22:50.979902 16663 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.195@KRBTEST.COM'} at 127.3.211.195:33141
I20260925 15:22:50.993763 3919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:22:51.015712 17151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:50.994792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52632 (local address 127.3.211.254:34085)
0925 15:22:51.002693 (+ 7901us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:51.002696 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:51.002716 (+ 20us) server_negotiation.cc:407] Connection header received
0925 15:22:51.002751 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:51.002753 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:51.002767 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:51.003684 (+ 917us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:51.003727 (+ 43us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:51.004575 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:51.005983 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:51.006369 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:51.006799 (+ 430us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:51.008210 (+ 1411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:51.008212 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:51.008212 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:51.008286 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:51.013948 (+ 5662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:51.014479 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:51.014480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:51.014480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:51.014546 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:51.014751 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:51.014751 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:51.014752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:51.014913 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:51.015467 (+ 554us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:51.015510 (+ 43us) server_negotiation.cc:299] Negotiation successful
0925 15:22:51.015599 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7866,"thread_start_us":75,"threads_started":1}
I20260925 15:22:51.018437 16663 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52632:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:22:51.020841 16663 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:22:51.032027 17160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:51.024731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.194:36321 (local address 127.0.0.1:58322)
0925 15:22:51.025878 (+ 1147us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:51.025887 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:51.025926 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:51.026198 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:51.026202 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:51.026204 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:22:51.026274 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:51.026276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:51.027856 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:51.027857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:51.030519 (+ 2662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:51.030521 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:51.031677 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:51.031678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:51.031772 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:51.031786 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 15:22:51.031855 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 15:22:51.031916 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1133,"thread_start_us":65,"threads_started":1}
I20260925 15:22:51.032567 17163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:51.025055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58322 (local address 127.3.211.194:36321)
0925 15:22:51.025831 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:51.025837 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:51.025973 (+ 136us) server_negotiation.cc:407] Connection header received
0925 15:22:51.026010 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:51.026013 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:51.026028 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:51.026104 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:22:51.026347 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:51.027769 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:51.030616 (+ 2847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:51.031490 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:51.031568 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:51.031635 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:51.032394 (+ 759us) server_negotiation.cc:299] Negotiation successful
0925 15:22:51.032446 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":741,"thread_start_us":54,"threads_started":1}
I20260925 15:22:51.034003 16930 tablet_service.cc:1467] Processing CreateTablet for tablet d4cd6add943c46a1b04290277aac2e79 (DEFAULT_TABLE table=test-table [id=f3f86473969946798fac8c40ce2da62b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:22:51.034437 16930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4cd6add943c46a1b04290277aac2e79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:22:51.043401 17159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:51.024909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:33757 (local address 127.0.0.1:40386)
0925 15:22:51.030188 (+ 5279us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:51.030201 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:51.030264 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:51.033944 (+ 3680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:51.033946 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:51.033949 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:22:51.033999 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:51.034000 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:51.039428 (+ 5428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:51.039429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:51.040979 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:51.040982 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:51.043136 (+ 2154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:51.043138 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:51.043213 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:51.043220 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 15:22:51.043252 (+ 32us) client_negotiation.cc:240] Negotiation successful
0925 15:22:51.043303 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5255,"thread_start_us":74,"threads_started":1}
I20260925 15:22:51.044106 17161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:51.024458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.195:34233 (local address 127.0.0.1:35102)
0925 15:22:51.025558 (+ 1100us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:51.025602 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:51.025695 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:51.034381 (+ 8686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:51.034383 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:51.034385 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:22:51.034429 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:51.034431 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:51.037671 (+ 3240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:51.037673 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:51.039316 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:51.039320 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:51.043868 (+ 4548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:51.043869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:51.043940 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:51.043945 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:22:51.043973 (+ 28us) client_negotiation.cc:240] Negotiation successful
0925 15:22:51.044033 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1068,"thread_start_us":91,"threads_started":1}
I20260925 15:22:51.044718 17162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:51.025017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40386 (local address 127.3.211.193:33757)
0925 15:22:51.033684 (+ 8667us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:51.033690 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:51.033702 (+ 12us) server_negotiation.cc:407] Connection header received
0925 15:22:51.033741 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:51.033743 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:51.033760 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:51.033850 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:22:51.035854 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:51.037354 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:51.041540 (+ 4186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:51.043048 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:51.044453 (+ 1405us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:51.044557 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:51.044589 (+ 32us) server_negotiation.cc:299] Negotiation successful
0925 15:22:51.044641 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8629,"thread_start_us":67,"threads_started":1}
I20260925 15:22:51.045185 17158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:51.024486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35102 (local address 127.3.211.195:34233)
0925 15:22:51.034107 (+ 9621us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:51.034111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:51.034121 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:22:51.034155 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:51.034157 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:51.034172 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:51.034587 (+ 415us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:22:51.034674 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:51.035691 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:51.041100 (+ 5409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:51.042945 (+ 1845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:51.044938 (+ 1993us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:51.045032 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:51.045060 (+ 28us) server_negotiation.cc:299] Negotiation successful
0925 15:22:51.045109 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9092,"thread_start_us":69,"threads_started":1}
I20260925 15:22:51.046250 17164 tablet_bootstrap.cc:492] T d4cd6add943c46a1b04290277aac2e79 P 0adb59e3f7b242c29e58a71dc004b245: Bootstrap starting.
I20260925 15:22:51.046154 17062 tablet_service.cc:1467] Processing CreateTablet for tablet d4cd6add943c46a1b04290277aac2e79 (DEFAULT_TABLE table=test-table [id=f3f86473969946798fac8c40ce2da62b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:22:51.046150 16796 tablet_service.cc:1467] Processing CreateTablet for tablet d4cd6add943c46a1b04290277aac2e79 (DEFAULT_TABLE table=test-table [id=f3f86473969946798fac8c40ce2da62b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:22:51.046536 16796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4cd6add943c46a1b04290277aac2e79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:22:51.046545 17062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4cd6add943c46a1b04290277aac2e79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:22:51.047142 17164 tablet_bootstrap.cc:654] T d4cd6add943c46a1b04290277aac2e79 P 0adb59e3f7b242c29e58a71dc004b245: Neither blocks nor log segments found. Creating new log.
I20260925 15:22:51.047518 17164 log.cc:824] T d4cd6add943c46a1b04290277aac2e79 P 0adb59e3f7b242c29e58a71dc004b245: Log is configured to *not* fsync() on all Append() calls
I20260925 15:22:51.048461 17164 tablet_bootstrap.cc:492] T d4cd6add943c46a1b04290277aac2e79 P 0adb59e3f7b242c29e58a71dc004b245: No bootstrap required, opened a new log
I20260925 15:22:51.048537 17164 ts_tablet_manager.cc:1397] T d4cd6add943c46a1b04290277aac2e79 P 0adb59e3f7b242c29e58a71dc004b245: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:22:51.051295 17164 raft_consensus.cc:348] T d4cd6add943c46a1b04290277aac2e79 P 0adb59e3f7b242c29e58a71dc004b245 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0adb59e3f7b242c29e58a71dc004b245" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 36321 } } peers { permanent_uuid: "2184196ed67a4d1fbbdfa9a57d23efa7" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 33757 } } peers { permanent_uuid: "eed3118981c64bb891264464ec90f208" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 34233 } }
I20260925 15:22:51.051473 17164 raft_consensus.cc:374] T d4cd6add943c46a1b04290277aac2e79 P 0adb59e3f7b242c29e58a71dc004b245 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:22:51.051502 17164 raft_consensus.cc:729] T d4cd6add943c46a1b04290277aac2e79 P 0adb59e3f7b242c29e58a71dc004b245 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0adb59e3f7b242c29e58a71dc004b245, State: Initialized, Role: FOLLOWER
I20260925 15:22:51.052035 17164 consensus_queue.cc:260] T d4cd6add943c46a1b04290277aac2e79 P 0adb59e3f7b242c29e58a71dc004b245 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0adb59e3f7b242c29e58a71dc004b245" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 36321 } } peers { permanent_uuid: "2184196ed67a4d1fbbdfa9a57d23efa7" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 33757 } } peers { permanent_uuid: "eed3118981c64bb891264464ec90f208" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 34233 } }
I20260925 15:22:51.052879 17164 ts_tablet_manager.cc:1428] T d4cd6add943c46a1b04290277aac2e79 P 0adb59e3f7b242c29e58a71dc004b245: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260925 15:22:51.053056 16996 heartbeater.cc:503] Master 127.3.211.254:34085 was elected leader, sending a full tablet report...
I20260925 15:22:51.054437 17167 tablet_bootstrap.cc:492] T d4cd6add943c46a1b04290277aac2e79 P eed3118981c64bb891264464ec90f208: Bootstrap starting.
I20260925 15:22:51.055199 17167 tablet_bootstrap.cc:654] T d4cd6add943c46a1b04290277aac2e79 P eed3118981c64bb891264464ec90f208: Neither blocks nor log segments found. Creating new log.
I20260925 15:22:51.055584 17167 log.cc:824] T d4cd6add943c46a1b04290277aac2e79 P eed3118981c64bb891264464ec90f208: Log is configured to *not* fsync() on all Append() calls
I20260925 15:22:51.056488 17167 tablet_bootstrap.cc:492] T d4cd6add943c46a1b04290277aac2e79 P eed3118981c64bb891264464ec90f208: No bootstrap required, opened a new log
I20260925 15:22:51.056558 17167 ts_tablet_manager.cc:1397] T d4cd6add943c46a1b04290277aac2e79 P eed3118981c64bb891264464ec90f208: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:22:51.057049 17166 tablet_bootstrap.cc:492] T d4cd6add943c46a1b04290277aac2e79 P 2184196ed67a4d1fbbdfa9a57d23efa7: Bootstrap starting.
I20260925 15:22:51.057760 17166 tablet_bootstrap.cc:654] T d4cd6add943c46a1b04290277aac2e79 P 2184196ed67a4d1fbbdfa9a57d23efa7: Neither blocks nor log segments found. Creating new log.
I20260925 15:22:51.058085 17166 log.cc:824] T d4cd6add943c46a1b04290277aac2e79 P 2184196ed67a4d1fbbdfa9a57d23efa7: Log is configured to *not* fsync() on all Append() calls
I20260925 15:22:51.058892 17167 raft_consensus.cc:348] T d4cd6add943c46a1b04290277aac2e79 P eed3118981c64bb891264464ec90f208 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0adb59e3f7b242c29e58a71dc004b245" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 36321 } } peers { permanent_uuid: "2184196ed67a4d1fbbdfa9a57d23efa7" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 33757 } } peers { permanent_uuid: "eed3118981c64bb891264464ec90f208" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 34233 } }
I20260925 15:22:51.059085 17167 raft_consensus.cc:374] T d4cd6add943c46a1b04290277aac2e79 P eed3118981c64bb891264464ec90f208 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:22:51.059118 17167 raft_consensus.cc:729] T d4cd6add943c46a1b04290277aac2e79 P eed3118981c64bb891264464ec90f208 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eed3118981c64bb891264464ec90f208, State: Initialized, Role: FOLLOWER
I20260925 15:22:51.059417 17167 consensus_queue.cc:260] T d4cd6add943c46a1b04290277aac2e79 P eed3118981c64bb891264464ec90f208 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0adb59e3f7b242c29e58a71dc004b245" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 36321 } } peers { permanent_uuid: "2184196ed67a4d1fbbdfa9a57d23efa7" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 33757 } } peers { permanent_uuid: "eed3118981c64bb891264464ec90f208" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 34233 } }
I20260925 15:22:51.059770 17167 ts_tablet_manager.cc:1428] T d4cd6add943c46a1b04290277aac2e79 P eed3118981c64bb891264464ec90f208: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:22:51.059929 17130 heartbeater.cc:503] Master 127.3.211.254:34085 was elected leader, sending a full tablet report...
I20260925 15:22:51.062242 17166 tablet_bootstrap.cc:492] T d4cd6add943c46a1b04290277aac2e79 P 2184196ed67a4d1fbbdfa9a57d23efa7: No bootstrap required, opened a new log
I20260925 15:22:51.062393 17166 ts_tablet_manager.cc:1397] T d4cd6add943c46a1b04290277aac2e79 P 2184196ed67a4d1fbbdfa9a57d23efa7: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260925 15:22:51.064833 17166 raft_consensus.cc:348] T d4cd6add943c46a1b04290277aac2e79 P 2184196ed67a4d1fbbdfa9a57d23efa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0adb59e3f7b242c29e58a71dc004b245" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 36321 } } peers { permanent_uuid: "2184196ed67a4d1fbbdfa9a57d23efa7" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 33757 } } peers { permanent_uuid: "eed3118981c64bb891264464ec90f208" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 34233 } }
I20260925 15:22:51.065140 17166 raft_consensus.cc:374] T d4cd6add943c46a1b04290277aac2e79 P 2184196ed67a4d1fbbdfa9a57d23efa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:22:51.065223 17166 raft_consensus.cc:729] T d4cd6add943c46a1b04290277aac2e79 P 2184196ed67a4d1fbbdfa9a57d23efa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2184196ed67a4d1fbbdfa9a57d23efa7, State: Initialized, Role: FOLLOWER
I20260925 15:22:51.065433 17166 consensus_queue.cc:260] T d4cd6add943c46a1b04290277aac2e79 P 2184196ed67a4d1fbbdfa9a57d23efa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0adb59e3f7b242c29e58a71dc004b245" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 36321 } } peers { permanent_uuid: "2184196ed67a4d1fbbdfa9a57d23efa7" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 33757 } } peers { permanent_uuid: "eed3118981c64bb891264464ec90f208" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 34233 } }
I20260925 15:22:51.065922 17166 ts_tablet_manager.cc:1428] T d4cd6add943c46a1b04290277aac2e79 P 2184196ed67a4d1fbbdfa9a57d23efa7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:22:51.066111 16862 heartbeater.cc:503] Master 127.3.211.254:34085 was elected leader, sending a full tablet report...
I20260925 15:22:51.206868 17172 raft_consensus.cc:482] T d4cd6add943c46a1b04290277aac2e79 P 2184196ed67a4d1fbbdfa9a57d23efa7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:22:51.207155 17172 raft_consensus.cc:504] T d4cd6add943c46a1b04290277aac2e79 P 2184196ed67a4d1fbbdfa9a57d23efa7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0adb59e3f7b242c29e58a71dc004b245" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 36321 } } peers { permanent_uuid: "2184196ed67a4d1fbbdfa9a57d23efa7" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 33757 } } peers { permanent_uuid: "eed3118981c64bb891264464ec90f208" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 34233 } }
I20260925 15:22:51.209246 17172 leader_election.cc:288] T d4cd6add943c46a1b04290277aac2e79 P 2184196ed67a4d1fbbdfa9a57d23efa7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0adb59e3f7b242c29e58a71dc004b245 (127.3.211.194:36321), eed3118981c64bb891264464ec90f208 (127.3.211.195:34233)
I20260925 15:22:51.213057 17163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:51.208548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:46657 (local address 127.3.211.194:36321)
0925 15:22:51.208601 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:51.208603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:51.209567 (+ 964us) server_negotiation.cc:407] Connection header received
0925 15:22:51.209683 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:51.209684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:51.209694 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:51.209724 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:22:51.209942 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:51.210664 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:51.212093 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:51.212643 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:51.212692 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:51.212701 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:51.212926 (+ 225us) server_negotiation.cc:299] Negotiation successful
0925 15:22:51.212971 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 15:22:51.213527 17173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:51.207867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.194:36321 (local address 127.3.211.193:46657)
0925 15:22:51.209482 (+ 1615us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:51.209496 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:51.209602 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:51.209794 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:51.209795 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:51.209797 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:22:51.209874 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:51.209876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:51.210772 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:51.210773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:51.211965 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:51.211970 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:51.212777 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:51.212778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:51.212845 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:51.212849 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:22:51.213382 (+ 533us) client_negotiation.cc:240] Negotiation successful
0925 15:22:51.213433 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1596,"thread_start_us":129,"threads_started":1}
I20260925 15:22:51.214007 16950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4cd6add943c46a1b04290277aac2e79" candidate_uuid: "2184196ed67a4d1fbbdfa9a57d23efa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0adb59e3f7b242c29e58a71dc004b245" is_pre_election: true
I20260925 15:22:51.214222 16950 raft_consensus.cc:2454] T d4cd6add943c46a1b04290277aac2e79 P 0adb59e3f7b242c29e58a71dc004b245 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2184196ed67a4d1fbbdfa9a57d23efa7 in term 0.
W20260925 15:22:51.214450 17131 tablet.cc:2369] T d4cd6add943c46a1b04290277aac2e79 P eed3118981c64bb891264464ec90f208: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:22:51.214818 16752 leader_election.cc:302] T d4cd6add943c46a1b04290277aac2e79 P 2184196ed67a4d1fbbdfa9a57d23efa7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0adb59e3f7b242c29e58a71dc004b245, 2184196ed67a4d1fbbdfa9a57d23efa7; no voters:
I20260925 15:22:51.215291 17172 raft_consensus.cc:2781] T d4cd6add943c46a1b04290277aac2e79 P 2184196ed67a4d1fbbdfa9a57d23efa7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:22:51.215353 17172 raft_consensus.cc:482] T d4cd6add943c46a1b04290277aac2e79 P 2184196ed67a4d1fbbdfa9a57d23efa7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:22:51.215373 17172 raft_consensus.cc:3037] T d4cd6add943c46a1b04290277aac2e79 P 2184196ed67a4d1fbbdfa9a57d23efa7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:22:51.216250 17172 raft_consensus.cc:504] T d4cd6add943c46a1b04290277aac2e79 P 2184196ed67a4d1fbbdfa9a57d23efa7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0adb59e3f7b242c29e58a71dc004b245" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 36321 } } peers { permanent_uuid: "2184196ed67a4d1fbbdfa9a57d23efa7" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 33757 } } peers { permanent_uuid: "eed3118981c64bb891264464ec90f208" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 34233 } }
I20260925 15:22:51.216694 16950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4cd6add943c46a1b04290277aac2e79" candidate_uuid: "2184196ed67a4d1fbbdfa9a57d23efa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0adb59e3f7b242c29e58a71dc004b245"
I20260925 15:22:51.216778 16950 raft_consensus.cc:3037] T d4cd6add943c46a1b04290277aac2e79 P 0adb59e3f7b242c29e58a71dc004b245 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:22:51.217660 16950 raft_consensus.cc:2454] T d4cd6add943c46a1b04290277aac2e79 P 0adb59e3f7b242c29e58a71dc004b245 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2184196ed67a4d1fbbdfa9a57d23efa7 in term 1.
I20260925 15:22:51.217888 16752 leader_election.cc:302] T d4cd6add943c46a1b04290277aac2e79 P 2184196ed67a4d1fbbdfa9a57d23efa7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0adb59e3f7b242c29e58a71dc004b245, 2184196ed67a4d1fbbdfa9a57d23efa7; no voters:
I20260925 15:22:51.218204 17172 leader_election.cc:288] T d4cd6add943c46a1b04290277aac2e79 P 2184196ed67a4d1fbbdfa9a57d23efa7 [CANDIDATE]: Term 1 election: Requested vote from peers 0adb59e3f7b242c29e58a71dc004b245 (127.3.211.194:36321), eed3118981c64bb891264464ec90f208 (127.3.211.195:34233)
I20260925 15:22:51.218293 17172 raft_consensus.cc:2781] T d4cd6add943c46a1b04290277aac2e79 P 2184196ed67a4d1fbbdfa9a57d23efa7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:22:51.218366 17172 raft_consensus.cc:686] T d4cd6add943c46a1b04290277aac2e79 P 2184196ed67a4d1fbbdfa9a57d23efa7 [term 1 LEADER]: Becoming Leader. State: Replica: 2184196ed67a4d1fbbdfa9a57d23efa7, State: Running, Role: LEADER
I20260925 15:22:51.218489 17172 consensus_queue.cc:237] T d4cd6add943c46a1b04290277aac2e79 P 2184196ed67a4d1fbbdfa9a57d23efa7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0adb59e3f7b242c29e58a71dc004b245" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 36321 } } peers { permanent_uuid: "2184196ed67a4d1fbbdfa9a57d23efa7" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 33757 } } peers { permanent_uuid: "eed3118981c64bb891264464ec90f208" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 34233 } }
I20260925 15:22:51.219446 16663 catalog_manager.cc:5495] T d4cd6add943c46a1b04290277aac2e79 P 2184196ed67a4d1fbbdfa9a57d23efa7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2184196ed67a4d1fbbdfa9a57d23efa7 (127.3.211.193). New cstate: current_term: 1 leader_uuid: "2184196ed67a4d1fbbdfa9a57d23efa7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0adb59e3f7b242c29e58a71dc004b245" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 36321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2184196ed67a4d1fbbdfa9a57d23efa7" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 33757 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eed3118981c64bb891264464ec90f208" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 34233 } health_report { overall_health: UNKNOWN } } }
I20260925 15:22:51.222239 17158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:51.209180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:33447 (local address 127.3.211.195:34233)
0925 15:22:51.209222 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:51.209224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:51.213940 (+ 4716us) server_negotiation.cc:407] Connection header received
0925 15:22:51.214059 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:51.214061 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:51.214072 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:51.214104 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:22:51.214456 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:51.215119 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:51.221295 (+ 6176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:51.221851 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:51.221903 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:51.221913 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:51.222121 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 15:22:51.222163 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 15:22:51.222672 17173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:51.208954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.195:34233 (local address 127.3.211.193:33447)
0925 15:22:51.213881 (+ 4927us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:51.213895 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:51.213970 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:51.214252 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:51.214253 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:51.214255 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:22:51.214367 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:51.214370 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:51.220028 (+ 5658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:51.220029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:51.221166 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:51.221171 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:51.221972 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:51.221973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:51.222032 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:51.222035 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:22:51.222549 (+ 514us) client_negotiation.cc:240] Negotiation successful
0925 15:22:51.222603 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4913,"thread_start_us":100,"threads_started":1}
I20260925 15:22:51.223168 17084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4cd6add943c46a1b04290277aac2e79" candidate_uuid: "2184196ed67a4d1fbbdfa9a57d23efa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eed3118981c64bb891264464ec90f208" is_pre_election: true
I20260925 15:22:51.223347 17084 raft_consensus.cc:2454] T d4cd6add943c46a1b04290277aac2e79 P eed3118981c64bb891264464ec90f208 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2184196ed67a4d1fbbdfa9a57d23efa7 in term 0.
I20260925 15:22:51.223484 17065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4cd6add943c46a1b04290277aac2e79" candidate_uuid: "2184196ed67a4d1fbbdfa9a57d23efa7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eed3118981c64bb891264464ec90f208"
I20260925 15:22:51.223567 17065 raft_consensus.cc:3037] T d4cd6add943c46a1b04290277aac2e79 P eed3118981c64bb891264464ec90f208 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:22:51.224500 17065 raft_consensus.cc:2454] T d4cd6add943c46a1b04290277aac2e79 P eed3118981c64bb891264464ec90f208 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2184196ed67a4d1fbbdfa9a57d23efa7 in term 1.
I20260925 15:22:51.257929 17162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:51.254570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40396 (local address 127.3.211.193:33757)
0925 15:22:51.254614 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:51.254616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:51.254623 (+ 7us) server_negotiation.cc:407] Connection header received
0925 15:22:51.254667 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:51.254668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:51.254690 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:51.254753 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:22:51.254934 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:51.255700 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:51.256827 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:51.257133 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:51.257192 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:51.257378 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:22:51.257530 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:22:51.257603 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:51.257821 (+ 218us) server_negotiation.cc:299] Negotiation successful
0925 15:22:51.257844 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 15:22:51.259847 17065 raft_consensus.cc:1261] T d4cd6add943c46a1b04290277aac2e79 P eed3118981c64bb891264464ec90f208 [term 1 FOLLOWER]: Refusing update from remote peer 2184196ed67a4d1fbbdfa9a57d23efa7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:22:51.260207 17175 consensus_queue.cc:1035] T d4cd6add943c46a1b04290277aac2e79 P 2184196ed67a4d1fbbdfa9a57d23efa7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eed3118981c64bb891264464ec90f208" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 34233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:22:51.261428 16950 raft_consensus.cc:1261] T d4cd6add943c46a1b04290277aac2e79 P 0adb59e3f7b242c29e58a71dc004b245 [term 1 FOLLOWER]: Refusing update from remote peer 2184196ed67a4d1fbbdfa9a57d23efa7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:22:51.262470 17172 consensus_queue.cc:1035] T d4cd6add943c46a1b04290277aac2e79 P 2184196ed67a4d1fbbdfa9a57d23efa7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0adb59e3f7b242c29e58a71dc004b245" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 36321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:22:51.265872 16663 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:22:51.265990 16663 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:22:51.266706 16663 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b9426d03e35647be9b6fea84c8c8ee75: Sending DeleteTablet for 3 replicas of tablet d4cd6add943c46a1b04290277aac2e79
I20260925 15:22:51.267129 16930 tablet_service.cc:1514] Processing DeleteTablet for tablet d4cd6add943c46a1b04290277aac2e79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:22:51 UTC) from {username='kudu', principal='kudu/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:58322
I20260925 15:22:51.267422 16796 tablet_service.cc:1514] Processing DeleteTablet for tablet d4cd6add943c46a1b04290277aac2e79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:22:51 UTC) from {username='kudu', principal='kudu/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:40386
I20260925 15:22:51.268131 17062 tablet_service.cc:1514] Processing DeleteTablet for tablet d4cd6add943c46a1b04290277aac2e79 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:22:51 UTC) from {username='kudu', principal='kudu/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:35102
I20260925 15:22:51.268396 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 16702
I20260925 15:22:51.268426 17178 mvcc.cc:205] Tried to move back new op lower bound from 7333272663077220352 to 7333272662911885312. Current Snapshot: MvccSnapshot[applied={T|T < 7333272663077220352 or (T in {7333272663077220352})}]
I20260925 15:22:51.268579 17180 mvcc.cc:205] Tried to move back new op lower bound from 7333272663077220352 to 7333272662911885312. Current Snapshot: MvccSnapshot[applied={T|T < 7333272663077220352 or (T in {7333272663077220352})}]
I20260925 15:22:51.269727 17185 mvcc.cc:205] Tried to move back new op lower bound from 7333272663077220352 to 7333272662911885312. Current Snapshot: MvccSnapshot[applied={T|T < 7333272663077220352}]
I20260925 15:22:51.272866 17189 tablet_replica.cc:330] T d4cd6add943c46a1b04290277aac2e79 P 0adb59e3f7b242c29e58a71dc004b245: stopping tablet replica
I20260925 15:22:51.273025 17189 raft_consensus.cc:2229] T d4cd6add943c46a1b04290277aac2e79 P 0adb59e3f7b242c29e58a71dc004b245 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:22:51.273111 17189 raft_consensus.cc:2258] T d4cd6add943c46a1b04290277aac2e79 P 0adb59e3f7b242c29e58a71dc004b245 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:22:51.273299 17191 tablet_replica.cc:330] T d4cd6add943c46a1b04290277aac2e79 P eed3118981c64bb891264464ec90f208: stopping tablet replica
I20260925 15:22:51.273411 17191 raft_consensus.cc:2229] T d4cd6add943c46a1b04290277aac2e79 P eed3118981c64bb891264464ec90f208 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:22:51.273530 17191 raft_consensus.cc:2258] T d4cd6add943c46a1b04290277aac2e79 P eed3118981c64bb891264464ec90f208 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:22:51.274129 17191 ts_tablet_manager.cc:1905] T d4cd6add943c46a1b04290277aac2e79 P eed3118981c64bb891264464ec90f208: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:22:51.276413 17191 ts_tablet_manager.cc:1918] T d4cd6add943c46a1b04290277aac2e79 P eed3118981c64bb891264464ec90f208: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:22:51.276510 17191 log.cc:1196] T d4cd6add943c46a1b04290277aac2e79 P eed3118981c64bb891264464ec90f208: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-2/wal/wals/d4cd6add943c46a1b04290277aac2e79
I20260925 15:22:51.276798 17191 ts_tablet_manager.cc:1939] T d4cd6add943c46a1b04290277aac2e79 P eed3118981c64bb891264464ec90f208: Deleting consensus metadata
I20260925 15:22:51.277293 16649 catalog_manager.cc:4841] TS eed3118981c64bb891264464ec90f208 (127.3.211.195:34233): tablet d4cd6add943c46a1b04290277aac2e79 (table test-table [id=f3f86473969946798fac8c40ce2da62b]) successfully deleted
I20260925 15:22:51.278520 17189 ts_tablet_manager.cc:1905] T d4cd6add943c46a1b04290277aac2e79 P 0adb59e3f7b242c29e58a71dc004b245: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:22:51.281594 17189 ts_tablet_manager.cc:1918] T d4cd6add943c46a1b04290277aac2e79 P 0adb59e3f7b242c29e58a71dc004b245: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:22:51.281762 17189 log.cc:1196] T d4cd6add943c46a1b04290277aac2e79 P 0adb59e3f7b242c29e58a71dc004b245: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790349610595785-3919-0/minicluster-data/ts-1/wal/wals/d4cd6add943c46a1b04290277aac2e79
I20260925 15:22:51.282146 17189 ts_tablet_manager.cc:1939] T d4cd6add943c46a1b04290277aac2e79 P 0adb59e3f7b242c29e58a71dc004b245: Deleting consensus metadata
W20260925 15:22:51.282385 16885 connection.cc:535] server connection from 127.3.211.193:46657 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 15:22:51.282759 16651 catalog_manager.cc:4841] TS 0adb59e3f7b242c29e58a71dc004b245 (127.3.211.194:36321): tablet d4cd6add943c46a1b04290277aac2e79 (table test-table [id=f3f86473969946798fac8c40ce2da62b]) successfully deleted
W20260925 15:22:51.282974 16651 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.211.193:33757 (error 108)
I20260925 15:22:51.283595 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 16866
I20260925 15:22:51.289347 17161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:51.289178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:33757 (local address 127.0.0.1:40404)
0925 15:22:51.289245 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:51.289290 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.193:33757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 15:22:51.289510 16651 catalog_manager.cc:4584] TS 2184196ed67a4d1fbbdfa9a57d23efa7 (127.3.211.193:33757): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d4cd6add943c46a1b04290277aac2e79: Network error: Client connection negotiation failed: client connection to 127.3.211.193:33757: connect: Connection refused (error 111)
I20260925 15:22:51.291687 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 16999
Sep 25 15:22:51 dist-test-slave-4bk3 krb5kdc[15668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349770, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.254@KRBTEST.COM for HTTP/127.3.211.212@KRBTEST.COM
I20260925 15:22:51.298576 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 16631
I20260925 15:22:51.304881 3919 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 15:22:51.640628 3919 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 15:22:51.640733 3919 mini_ranger.cc:66] Stopping Ranger...
I20260925 15:22:51.986297 3919 mini_ranger.cc:68] Stopped Ranger
2026-09-25 15:22:51.986 UTC [15698] LOG: received smart shutdown request
2026-09-25 15:22:51.991 UTC [15698] LOG: background worker "logical replication launcher" (PID 15707) exited with exit code 1
2026-09-25 15:22:51.991 UTC [15702] LOG: shutting down
2026-09-25 15:22:52.178 UTC [15698] LOG: database system is shut down
2026-09-25T15:22:52Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53264 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:22:52 dist-test-slave-4bk3 krb5kdc[17202](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:22:52 dist-test-slave-4bk3 krb5kdc[17202](info): set up 2 sockets
Sep 25 15:22:52 dist-test-slave-4bk3 krb5kdc[17202](info): commencing operation
krb5kdc: starting...
W20260925 15:22:53.374346 3919 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Sep 25 15:22:53 dist-test-slave-4bk3 krb5kdc[17202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:22:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:22:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.211.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
I20260925 15:22:53.504878 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:35715
--webserver_interface=127.3.211.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:35495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.211.254:35715,127.3.211.253:39053,127.3.211.252:36881
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:22:53.601600 17215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:22:53.601861 17215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:22:53.601888 17215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:22:53.603307 17215 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:22:53.603361 17215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:22:53.603379 17215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:22:53.603394 17215 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:22:53.603436 17215 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:22:53.605369 17215 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:35495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.211.254:35715,127.3.211.253:39053,127.3.211.252:36881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:35715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.211.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:22:53.605644 17215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:22:53.605849 17215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:22:53.610901 17220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:53.610936 17221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:53.610891 17223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:22:53.611225 17215 server_base.cc:1034] running on GCE node
I20260925 15:22:53.611482 17215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:22:53.611795 17215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:22:53.612993 17215 hybrid_clock.cc:648] HybridClock initialized: now 1790349773612949 us; error 61 us; skew 500 ppm
Sep 25 15:22:53 dist-test-slave-4bk3 krb5kdc[17202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349773, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:22:53.615813 17215 init.cc:377] Logged in from keytab as kudu/127.3.211.254@KRBTEST.COM (short username kudu)
I20260925 15:22:53.616567 17215 webserver.cc:460] Webserver started at http://127.3.211.254:43999/ using document root <none> and password file <none>
I20260925 15:22:53.616856 17215 fs_manager.cc:359] Metadata directory not provided
I20260925 15:22:53.616945 17215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:22:53.617115 17215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:22:53.618276 17215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/minicluster-data/master-0/data/instance:
uuid: "7a55b8b55fb34b639002808e0d5c2c20"
format_stamp: "Formatted at 2026-09-25 15:22:53 on dist-test-slave-4bk3"
server_key: "c0085a16da94c8d2d1c1c810bf2490ce"
server_key_iv: "950a13a877108e89e87dc81a98e69b52"
server_key_version: "encryptionkey@0"
I20260925 15:22:53.618665 17215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/minicluster-data/master-0/wal/instance:
uuid: "7a55b8b55fb34b639002808e0d5c2c20"
format_stamp: "Formatted at 2026-09-25 15:22:53 on dist-test-slave-4bk3"
server_key: "c0085a16da94c8d2d1c1c810bf2490ce"
server_key_iv: "950a13a877108e89e87dc81a98e69b52"
server_key_version: "encryptionkey@0"
I20260925 15:22:53.620203 17215 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:22:53.621177 17230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:53.621429 17215 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:22:53.621532 17215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/minicluster-data/master-0/wal
uuid: "7a55b8b55fb34b639002808e0d5c2c20"
format_stamp: "Formatted at 2026-09-25 15:22:53 on dist-test-slave-4bk3"
server_key: "c0085a16da94c8d2d1c1c810bf2490ce"
server_key_iv: "950a13a877108e89e87dc81a98e69b52"
server_key_version: "encryptionkey@0"
I20260925 15:22:53.621623 17215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:22:53.634939 17215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:22:53.637482 17215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:22:53.637662 17215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:22:53.641954 17215 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.254:35715
I20260925 15:22:53.641992 17282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.254:35715 every 8 connection(s)
I20260925 15:22:53.642400 17215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
I20260925 15:22:53.643572 17283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:22:53.645319 17283 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.211.254" port: 35715 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:22:53.650842 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 17215
I20260925 15:22:53.650967 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/minicluster-data/master-0/wal/instance
I20260925 15:22:53.651204 3919 external_mini_cluster.cc:1428] Setting key ea22703cf0bee2f8fbebe23a950ebae4
Sep 25 15:22:53 dist-test-slave-4bk3 krb5kdc[17202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349773, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.254@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:22:53.657689 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:53.646335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44236 (local address 127.3.211.254:35715)
0925 15:22:53.646566 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:53.646579 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:53.646848 (+ 269us) server_negotiation.cc:407] Connection header received
0925 15:22:53.647805 (+ 957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:53.647809 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:53.647840 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:53.648019 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:53.648761 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:53.649663 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:53.650576 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:53.651071 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:53.651215 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:53.653863 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:53.653869 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:53.653869 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:53.653877 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:53.655765 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:53.656263 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:53.656266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:53.656267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:53.656458 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:53.656723 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:53.656724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:53.656725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:53.656960 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:53.657057 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:53.657238 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 15:22:53.657423 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":189,"thread_start_us":91,"threads_started":1}
I20260925 15:22:53.657698 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:53.646172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:35715 (local address 127.0.0.1:44236)
0925 15:22:53.646415 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:53.646566 (+ 151us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:53.647476 (+ 910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:53.648023 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:53.648026 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:53.648449 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:22:53.648643 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:53.648649 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:53.649779 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:53.649780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:53.650470 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:53.650471 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:53.651211 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:53.651212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:53.651327 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:53.651949 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:22:53.651958 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:22:53.653752 (+ 1794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:22:53.655869 (+ 2117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:53.655875 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:53.655878 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:53.656080 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:53.656550 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:53.656551 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:53.656551 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:53.656654 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:53.657034 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:22:53.657039 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:22:53.657101 (+ 62us) client_negotiation.cc:769] Sending connection context
0925 15:22:53.657215 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 15:22:53.657486 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":126,"thread_start_us":124,"threads_started":1}
WARNING: no policy specified for kudu/127.3.211.253@KRBTEST.COM; defaulting to no policy
I20260925 15:22:53.659094 17283 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.211.253" port: 39053 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:22:53.659482 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:53.659352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.253:39053 (local address 127.0.0.1:33584)
0925 15:22:53.659405 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:53.659452 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.253:39053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 15:22:53.659626 17233 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.211.253:39053: connect: Connection refused (error 111)
I20260925 15:22:53.660054 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:53.659959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.253:39053 (local address 127.0.0.1:33594)
0925 15:22:53.660008 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:53.660032 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.253:39053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 15:22:53.660167 17283 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.211.253:39053: Network error: Client connection negotiation failed: client connection to 127.3.211.253:39053: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.253.keytab.
Entry for principal kudu/127.3.211.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.253.keytab.
I20260925 15:22:53.677702 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.253:39053
--webserver_interface=127.3.211.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.253.keytab
--principal=kudu/127.3.211.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:35495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.211.254:35715,127.3.211.253:39053,127.3.211.252:36881
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:22:53.709434 17283 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.211.253" port: 39053 } attempt: 1
I20260925 15:22:53.709975 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:53.709828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.253:39053 (local address 127.0.0.1:33602)
0925 15:22:53.709908 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:53.709946 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.253:39053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 15:22:53.710335 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:53.710232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.253:39053 (local address 127.0.0.1:33604)
0925 15:22:53.710282 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:53.710315 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.253:39053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 15:22:53.710520 17283 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.211.253:39053: Network error: Client connection negotiation failed: client connection to 127.3.211.253:39053: connect: Connection refused (error 111)
W20260925 15:22:53.766480 17290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:22:53.766723 17290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:22:53.766750 17290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:22:53.768177 17290 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:22:53.768241 17290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:22:53.768256 17290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:22:53.768270 17290 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:22:53.768282 17290 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:22:53.770406 17290 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:35495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.211.254:35715,127.3.211.253:39053,127.3.211.252:36881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.253.keytab
--principal=kudu/127.3.211.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.253:39053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.211.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:22:53.770725 17290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:22:53.770977 17290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:22:53.776088 17298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:53.776183 17296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:53.776104 17295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:22:53.776563 17290 server_base.cc:1034] running on GCE node
I20260925 15:22:53.776914 17290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:22:53.777218 17290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:22:53.778451 17290 hybrid_clock.cc:648] HybridClock initialized: now 1790349773778414 us; error 64 us; skew 500 ppm
I20260925 15:22:53.778687 17283 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.211.253" port: 39053 } attempt: 2
I20260925 15:22:53.779093 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:53.778962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.253:39053 (local address 127.0.0.1:33614)
0925 15:22:53.779022 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:53.779057 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.253:39053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 15:22:53.779515 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:53.779383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.253:39053 (local address 127.0.0.1:33630)
0925 15:22:53.779453 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:53.779484 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.253:39053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 15:22:53.779650 17283 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.211.253:39053: Network error: Client connection negotiation failed: client connection to 127.3.211.253:39053: connect: Connection refused (error 111)
Sep 25 15:22:53 dist-test-slave-4bk3 krb5kdc[17202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349773, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:22:53.781567 17290 init.cc:377] Logged in from keytab as kudu/127.3.211.253@KRBTEST.COM (short username kudu)
I20260925 15:22:53.782073 17290 webserver.cc:460] Webserver started at http://127.3.211.253:34691/ using document root <none> and password file <none>
I20260925 15:22:53.782335 17290 fs_manager.cc:359] Metadata directory not provided
I20260925 15:22:53.782389 17290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:22:53.782567 17290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:22:53.783679 17290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/minicluster-data/master-1/data/instance:
uuid: "b3810ae8a6254c23914f448659e0a8a8"
format_stamp: "Formatted at 2026-09-25 15:22:53 on dist-test-slave-4bk3"
server_key: "382b9855138f0b8399b66a231e685d69"
server_key_iv: "044e0f8cafd3606ea4e178602d713a0b"
server_key_version: "encryptionkey@0"
I20260925 15:22:53.784056 17290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/minicluster-data/master-1/wal/instance:
uuid: "b3810ae8a6254c23914f448659e0a8a8"
format_stamp: "Formatted at 2026-09-25 15:22:53 on dist-test-slave-4bk3"
server_key: "382b9855138f0b8399b66a231e685d69"
server_key_iv: "044e0f8cafd3606ea4e178602d713a0b"
server_key_version: "encryptionkey@0"
I20260925 15:22:53.785625 17290 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:22:53.786638 17305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:53.786983 17290 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:22:53.787145 17290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/minicluster-data/master-1/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/minicluster-data/master-1/wal
uuid: "b3810ae8a6254c23914f448659e0a8a8"
format_stamp: "Formatted at 2026-09-25 15:22:53 on dist-test-slave-4bk3"
server_key: "382b9855138f0b8399b66a231e685d69"
server_key_iv: "044e0f8cafd3606ea4e178602d713a0b"
server_key_version: "encryptionkey@0"
I20260925 15:22:53.787241 17290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:22:53.797842 17290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:22:53.800428 17290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:22:53.800604 17290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:22:53.804819 17290 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.253:39053
I20260925 15:22:53.804857 17357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.253:39053 every 8 connection(s)
I20260925 15:22:53.805275 17290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/minicluster-data/master-1/data/info.pb
I20260925 15:22:53.806198 17358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:22:53.807863 17358 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.211.254" port: 35715 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:22:53.813457 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 17290
I20260925 15:22:53.813587 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/minicluster-data/master-1/wal/instance
I20260925 15:22:53.813822 3919 external_mini_cluster.cc:1428] Setting key 1201b27f39a521a9b39c400934427743
Sep 25 15:22:53 dist-test-slave-4bk3 krb5kdc[17202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349773, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.253@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:22:53.819108 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:53.808706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44246 (local address 127.3.211.254:35715)
0925 15:22:53.808790 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:53.808792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:53.808985 (+ 193us) server_negotiation.cc:407] Connection header received
0925 15:22:53.809520 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:53.809521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:53.809535 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:53.809601 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:53.810284 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:53.810868 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:53.811893 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:53.812114 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:53.812198 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:53.815043 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:53.815048 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:53.815049 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:53.815061 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:53.817259 (+ 2198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:53.817801 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:53.817803 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:53.817803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:53.817873 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:53.818272 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:53.818273 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:53.818273 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:53.818414 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:53.818478 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:53.818847 (+ 369us) server_negotiation.cc:299] Negotiation successful
0925 15:22:53.818960 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 15:22:53.819130 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:53.808658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:35715 (local address 127.0.0.1:44246)
0925 15:22:53.808858 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:53.808898 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:53.809397 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:53.809682 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:53.809686 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:53.809953 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:22:53.810174 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:53.810178 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:53.810961 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:53.810961 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:53.811800 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:53.811802 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:53.812217 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:53.812218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:53.812416 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:53.812979 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:22:53.812989 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:22:53.814833 (+ 1844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:22:53.817418 (+ 2585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:53.817431 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:53.817434 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:53.817709 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:53.817983 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:53.817984 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:53.817985 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:53.818173 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:53.818493 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:22:53.818500 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:22:53.818699 (+ 199us) client_negotiation.cc:769] Sending connection context
0925 15:22:53.818797 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 15:22:53.818946 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":108,"threads_started":1}
I20260925 15:22:53.819877 17358 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.211.253" port: 39053 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.211.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.252@KRBTEST.COM" created.
Sep 25 15:22:53 dist-test-slave-4bk3 krb5kdc[17202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349773, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.253@KRBTEST.COM for kudu/127.3.211.253@KRBTEST.COM
I20260925 15:22:53.829679 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:53.820205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.253:39053 (local address 127.0.0.1:33646)
0925 15:22:53.820265 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:53.820277 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:53.820323 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:53.820635 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:53.820636 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:53.820814 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:22:53.820886 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:53.820888 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:53.821860 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:53.821862 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:53.822374 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:53.822376 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:53.823041 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:53.823042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:53.823111 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:53.823624 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:22:53.823628 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:22:53.825839 (+ 2211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:22:53.828260 (+ 2421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:53.828262 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:53.828262 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:53.828552 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:53.828809 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:53.828809 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:53.828810 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:53.828867 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:53.829288 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:22:53.829289 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:22:53.829394 (+ 105us) client_negotiation.cc:769] Sending connection context
0925 15:22:53.829471 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 15:22:53.829589 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 15:22:53.829818 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:53.820273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33646 (local address 127.3.211.253:39053)
0925 15:22:53.820490 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:53.820495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:53.820513 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:22:53.820532 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:53.820534 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:53.820546 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:53.820636 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:53.820976 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:53.821726 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:53.822538 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:53.822953 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:53.823021 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:53.825956 (+ 2935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:53.825974 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:53.825975 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:53.825982 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:53.828120 (+ 2138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:53.828648 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:53.828652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:53.828652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:53.828725 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:53.828941 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:53.828942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:53.828942 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:53.829192 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:53.829290 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:53.829559 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 15:22:53.829694 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":9,"server-negotiator.queue_time_us":187,"thread_start_us":136,"threads_started":1}
I20260925 15:22:53.830485 17358 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.211.252" port: 36881 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:22:53.831210 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:53.830978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.252:36881 (local address 127.0.0.1:44142)
0925 15:22:53.831056 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:53.831157 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.252:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 15:22:53.831410 17307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.211.252:36881: connect: Connection refused (error 111)
I20260925 15:22:53.832096 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:53.831934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.252:36881 (local address 127.0.0.1:44158)
0925 15:22:53.832021 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:53.832056 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.252:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260925 15:22:53.832242 17358 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.211.252:36881: Network error: Client connection negotiation failed: client connection to 127.3.211.252:36881: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.252.keytab.
Entry for principal kudu/127.3.211.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.252.keytab.
I20260925 15:22:53.843654 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.252:36881
--webserver_interface=127.3.211.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.252.keytab
--principal=kudu/127.3.211.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:35495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.211.254:35715,127.3.211.253:39053,127.3.211.252:36881
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:22:53.870846 17283 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.211.253" port: 39053 } attempt: 3
Sep 25 15:22:53 dist-test-slave-4bk3 krb5kdc[17202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349773, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.254@KRBTEST.COM for kudu/127.3.211.253@KRBTEST.COM
I20260925 15:22:53.879819 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:53.871296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.253:39053 (local address 127.0.0.1:33648)
0925 15:22:53.871363 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:53.871378 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:53.871430 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:53.871645 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:53.871646 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:53.871805 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:22:53.871879 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:53.871881 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:53.872922 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:53.872923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:53.873479 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:53.873482 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:53.874017 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:53.874019 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:53.874087 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:53.874629 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:22:53.874634 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:22:53.876307 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:22:53.878456 (+ 2149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:53.878457 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:53.878458 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:53.878682 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:53.878996 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:53.878997 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:53.878998 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:53.879120 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:53.879506 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:22:53.879507 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:22:53.879563 (+ 56us) client_negotiation.cc:769] Sending connection context
0925 15:22:53.879639 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 15:22:53.879740 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 15:22:53.879863 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:53.871403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33648 (local address 127.3.211.253:39053)
0925 15:22:53.871480 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:53.871482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:53.871494 (+ 12us) server_negotiation.cc:407] Connection header received
0925 15:22:53.871527 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:53.871528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:53.871543 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:53.871632 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:53.871980 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:53.872786 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:53.873586 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:53.873917 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:53.873997 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:53.876528 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:53.876530 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:53.876531 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:53.876541 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:53.878278 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:53.878773 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:53.878775 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:53.878775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:53.878842 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:53.879220 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:53.879221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:53.879222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:53.879379 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:53.879464 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:53.879682 (+ 218us) server_negotiation.cc:299] Negotiation successful
0925 15:22:53.879771 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 15:22:53.880489 17283 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.211.252" port: 36881 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:22:53.880908 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:53.880790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.252:36881 (local address 127.0.0.1:44160)
0925 15:22:53.880845 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:53.880884 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.252:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 15:22:53.881338 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:53.881186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.252:36881 (local address 127.0.0.1:44170)
0925 15:22:53.881267 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:53.881314 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.252:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260925 15:22:53.881488 17358 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.211.252" port: 36881 } attempt: 1
W20260925 15:22:53.881577 17283 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.211.252:36881: Network error: Client connection negotiation failed: client connection to 127.3.211.252:36881: connect: Connection refused (error 111)
I20260925 15:22:53.881933 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:53.881805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.252:36881 (local address 127.0.0.1:44186)
0925 15:22:53.881867 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:53.881897 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.252:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 15:22:53.882493 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:53.882355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.252:36881 (local address 127.0.0.1:44198)
0925 15:22:53.882431 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:53.882465 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.252:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260925 15:22:53.882640 17358 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.211.252:36881: Network error: Client connection negotiation failed: client connection to 127.3.211.252:36881: connect: Connection refused (error 111)
I20260925 15:22:53.912817 17283 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.211.252" port: 36881 } attempt: 1
I20260925 15:22:53.913395 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:53.913234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.252:36881 (local address 127.0.0.1:44212)
0925 15:22:53.913310 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:53.913361 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.252:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 15:22:53.913789 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:53.913682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.252:36881 (local address 127.0.0.1:44216)
0925 15:22:53.913731 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:53.913764 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.252:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 15:22:53.913926 17283 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.211.252:36881: Network error: Client connection negotiation failed: client connection to 127.3.211.252:36881: connect: Connection refused (error 111)
W20260925 15:22:53.934844 17365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:22:53.935115 17365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:22:53.935142 17365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:22:53.936474 17365 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:22:53.936522 17365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:22:53.936537 17365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:22:53.936550 17365 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:22:53.936563 17365 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:22:53.938210 17365 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:35495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.211.254:35715,127.3.211.253:39053,127.3.211.252:36881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.252.keytab
--principal=kudu/127.3.211.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.252:36881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.211.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:22:53.938504 17365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:22:53.938704 17365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:22:53.943840 17370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:53.943835 17373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:22:53.943871 17365 server_base.cc:1034] running on GCE node
W20260925 15:22:53.943835 17371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:22:53.944414 17365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:22:53.944777 17365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:22:53.945955 17365 hybrid_clock.cc:648] HybridClock initialized: now 1790349773945945 us; error 37 us; skew 500 ppm
Sep 25 15:22:53 dist-test-slave-4bk3 krb5kdc[17202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349773, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:22:53.948801 17365 init.cc:377] Logged in from keytab as kudu/127.3.211.252@KRBTEST.COM (short username kudu)
I20260925 15:22:53.949347 17365 webserver.cc:460] Webserver started at http://127.3.211.252:45977/ using document root <none> and password file <none>
I20260925 15:22:53.949585 17365 fs_manager.cc:359] Metadata directory not provided
I20260925 15:22:53.949631 17365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:22:53.949774 17365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:22:53.950788 17358 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.211.252" port: 36881 } attempt: 2
I20260925 15:22:53.950925 17365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/minicluster-data/master-2/data/instance:
uuid: "9cd022b0a54244f7927b3fc6d40377c0"
format_stamp: "Formatted at 2026-09-25 15:22:53 on dist-test-slave-4bk3"
server_key: "32c88bf3a0dfc5b9e0a951cd1518e612"
server_key_iv: "2f762ea2e0ebd8e19b2d36746465282d"
server_key_version: "encryptionkey@0"
I20260925 15:22:53.951241 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:53.951104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.252:36881 (local address 127.0.0.1:44226)
0925 15:22:53.951171 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:53.951206 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.252:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 15:22:53.951403 17365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/minicluster-data/master-2/wal/instance:
uuid: "9cd022b0a54244f7927b3fc6d40377c0"
format_stamp: "Formatted at 2026-09-25 15:22:53 on dist-test-slave-4bk3"
server_key: "32c88bf3a0dfc5b9e0a951cd1518e612"
server_key_iv: "2f762ea2e0ebd8e19b2d36746465282d"
server_key_version: "encryptionkey@0"
I20260925 15:22:53.951632 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:53.951520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.252:36881 (local address 127.0.0.1:44236)
0925 15:22:53.951566 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:53.951604 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.252:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260925 15:22:53.951792 17358 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.211.252:36881: Network error: Client connection negotiation failed: client connection to 127.3.211.252:36881: connect: Connection refused (error 111)
I20260925 15:22:53.953099 17365 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:22:53.954234 17380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:53.954559 17365 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:22:53.954655 17365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/minicluster-data/master-2/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/minicluster-data/master-2/wal
uuid: "9cd022b0a54244f7927b3fc6d40377c0"
format_stamp: "Formatted at 2026-09-25 15:22:53 on dist-test-slave-4bk3"
server_key: "32c88bf3a0dfc5b9e0a951cd1518e612"
server_key_iv: "2f762ea2e0ebd8e19b2d36746465282d"
server_key_version: "encryptionkey@0"
I20260925 15:22:53.954773 17365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:22:53.979028 17365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:22:53.981917 17365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:22:53.982054 17365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:22:53.986539 17365 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.252:36881
I20260925 15:22:53.986640 17432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.252:36881 every 8 connection(s)
I20260925 15:22:53.986986 17365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/minicluster-data/master-2/data/info.pb
I20260925 15:22:53.987986 17433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:22:53.989075 17283 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.211.252" port: 36881 } attempt: 2
I20260925 15:22:53.989651 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 17365
I20260925 15:22:53.989742 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790349610595785-3919-0/minicluster-data/master-2/wal/instance
I20260925 15:22:53.989962 3919 external_mini_cluster.cc:1428] Setting key 18e2a1d98af5ef93ca837be73f32cc38
I20260925 15:22:53.990936 17433 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.211.254" port: 35715 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:22:53 dist-test-slave-4bk3 krb5kdc[17202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349773, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.254@KRBTEST.COM for kudu/127.3.211.252@KRBTEST.COM
Sep 25 15:22:53 dist-test-slave-4bk3 krb5kdc[17202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.211.252@KRBTEST.COM
Sep 25 15:22:53 dist-test-slave-4bk3 krb5kdc[17202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349773, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.252@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:22:54.000916 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:53.989452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.252:36881 (local address 127.0.0.1:44244)
0925 15:22:53.989520 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:53.989533 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:53.989584 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:53.990870 (+ 1286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:53.990871 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:53.991068 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:22:53.991137 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:53.991139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:53.992694 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:53.992695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:53.993489 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:53.993491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:53.994083 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:53.994084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:53.994149 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:53.994759 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:22:53.994766 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:22:53.996417 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:22:53.999057 (+ 2640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:53.999058 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:53.999058 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:53.999359 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:53.999674 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:53.999676 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:53.999676 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:53.999748 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:54.000445 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:22:54.000447 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:22:54.000507 (+ 60us) client_negotiation.cc:769] Sending connection context
0925 15:22:54.000634 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 15:22:54.000765 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":1}
I20260925 15:22:54.001436 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:53.991774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44258 (local address 127.3.211.254:35715)
0925 15:22:53.991821 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:53.991823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:53.992031 (+ 208us) server_negotiation.cc:407] Connection header received
0925 15:22:53.992080 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:53.992081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:53.992093 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:53.992144 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:53.992735 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:53.993472 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:53.994485 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:53.994824 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:53.994904 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:53.997924 (+ 3020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:53.997926 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:53.997926 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:53.997932 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:53.999740 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:54.000445 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:54.000473 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:54.000473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:54.000537 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:54.000775 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:54.000776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:54.000776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:54.000907 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:54.000966 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:54.001258 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 15:22:54.001346 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":24}
I20260925 15:22:54.001672 17435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:53.989752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44244 (local address 127.3.211.252:36881)
0925 15:22:53.989961 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:53.989969 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:53.990002 (+ 33us) server_negotiation.cc:407] Connection header received
0925 15:22:53.990644 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:53.990654 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:53.990727 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:53.990886 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:53.991248 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:53.992607 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:53.993575 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:53.993972 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:53.994089 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:53.996546 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:53.996553 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:53.996553 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:53.996560 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:53.998770 (+ 2210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:53.999437 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:53.999441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:53.999442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:53.999571 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:53.999883 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:53.999884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:53.999884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:54.000309 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:54.001182 (+ 873us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:54.001239 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 15:22:54.001394 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":107,"threads_started":1}
I20260925 15:22:54.001804 17438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:53.991706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:35715 (local address 127.0.0.1:44258)
0925 15:22:53.991903 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:53.991949 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:53.992025 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:53.992160 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:53.992162 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:53.992543 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:22:53.992629 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:53.992630 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:53.993557 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:53.993558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:53.994403 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:53.994405 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:53.994919 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:53.994921 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:53.994998 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:53.995471 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:22:53.995484 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:22:53.997816 (+ 2332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:22:53.999842 (+ 2026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:53.999850 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:53.999853 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:54.000308 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:54.000626 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:54.000627 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:54.000628 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:54.000693 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:54.001005 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:22:54.001013 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:22:54.001098 (+ 85us) client_negotiation.cc:769] Sending connection context
0925 15:22:54.001534 (+ 436us) client_negotiation.cc:240] Negotiation successful
0925 15:22:54.001654 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":3,"thread_start_us":86,"threads_started":1}
I20260925 15:22:54.002089 17437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:53.990381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44246 (local address 127.3.211.252:36881)
0925 15:22:53.990509 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:53.990511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:53.990658 (+ 147us) server_negotiation.cc:407] Connection header received
0925 15:22:53.990706 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:53.990707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:53.990733 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:53.990886 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:53.991774 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:53.992614 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:53.993427 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:53.993976 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:53.994082 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:53.997583 (+ 3501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:53.997584 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:53.997585 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:53.997592 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:54.000322 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:54.000751 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:54.000752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:54.000752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:54.000808 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:54.001149 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:54.001150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:54.001150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:54.001288 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:54.001452 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:54.001822 (+ 370us) server_negotiation.cc:299] Negotiation successful
0925 15:22:54.001924 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":67,"threads_started":1}
I20260925 15:22:54.002616 17433 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.211.253" port: 39053 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:22:54.004191 17283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a55b8b55fb34b639002808e0d5c2c20: Bootstrap starting.
I20260925 15:22:54.005113 17283 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a55b8b55fb34b639002808e0d5c2c20: Neither blocks nor log segments found. Creating new log.
I20260925 15:22:54.005452 17283 log.cc:824] T 00000000000000000000000000000000 P 7a55b8b55fb34b639002808e0d5c2c20: Log is configured to *not* fsync() on all Append() calls
I20260925 15:22:54.006734 17283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a55b8b55fb34b639002808e0d5c2c20: No bootstrap required, opened a new log
Sep 25 15:22:54 dist-test-slave-4bk3 krb5kdc[17202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
Sep 25 15:22:54 dist-test-slave-4bk3 krb5kdc[17202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349773, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.252@KRBTEST.COM for kudu/127.3.211.253@KRBTEST.COM
I20260925 15:22:54.009124 17283 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a55b8b55fb34b639002808e0d5c2c20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a55b8b55fb34b639002808e0d5c2c20" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 35715 } } peers { permanent_uuid: "b3810ae8a6254c23914f448659e0a8a8" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 39053 } } peers { permanent_uuid: "9cd022b0a54244f7927b3fc6d40377c0" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 36881 } }
I20260925 15:22:54.009336 17283 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a55b8b55fb34b639002808e0d5c2c20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:22:54.009402 17283 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a55b8b55fb34b639002808e0d5c2c20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a55b8b55fb34b639002808e0d5c2c20, State: Initialized, Role: FOLLOWER
I20260925 15:22:54.009610 17283 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a55b8b55fb34b639002808e0d5c2c20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a55b8b55fb34b639002808e0d5c2c20" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 35715 } } peers { permanent_uuid: "b3810ae8a6254c23914f448659e0a8a8" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 39053 } } peers { permanent_uuid: "9cd022b0a54244f7927b3fc6d40377c0" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 36881 } }
I20260925 15:22:54.010057 17283 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a55b8b55fb34b639002808e0d5c2c20 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 15:22:54.012311 17451 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7a55b8b55fb34b639002808e0d5c2c20: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:22:54.012504 17451 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:22:54.012791 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:54.003034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44262 (local address 127.3.211.254:35715)
0925 15:22:54.003070 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:54.003072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:54.003158 (+ 86us) server_negotiation.cc:407] Connection header received
0925 15:22:54.003213 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:54.003213 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:54.003223 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:54.003278 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:54.003781 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:54.004258 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:54.005038 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:54.005298 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:54.005361 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:54.008143 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:54.008145 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:54.008145 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:54.008154 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:54.010221 (+ 2067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:54.011424 (+ 1203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:54.011426 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:54.011426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:54.011514 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:54.011765 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:54.011766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:54.011766 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:54.011906 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:54.012075 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:54.012606 (+ 531us) server_negotiation.cc:299] Negotiation successful
0925 15:22:54.012698 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 15:22:54.013082 17438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:54.002907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.253:39053 (local address 127.0.0.1:33654)
0925 15:22:54.002954 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:54.002965 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:54.003003 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:54.003216 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:54.003219 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:54.003422 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:22:54.003559 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:54.003562 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:54.004588 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:54.004589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:54.005467 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:54.005471 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:54.006000 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:54.006001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:54.006090 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:54.006562 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:22:54.006569 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:22:54.008579 (+ 2010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:22:54.011117 (+ 2538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:54.011118 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:54.011119 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:54.011350 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:54.012023 (+ 673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:54.012026 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:54.012027 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:54.012211 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:54.012667 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:22:54.012669 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:22:54.012734 (+ 65us) client_negotiation.cc:769] Sending connection context
0925 15:22:54.012788 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 15:22:54.012965 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 15:22:54.014138 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:54.002997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33654 (local address 127.3.211.253:39053)
0925 15:22:54.003044 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:54.003046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:54.003056 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:22:54.003082 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:54.003083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:54.003095 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:54.003157 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:54.003668 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:54.004469 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:54.005640 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:54.005888 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:54.005960 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:54.008704 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:54.008707 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:54.008707 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:54.008715 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:54.010993 (+ 2278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:54.011639 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:54.011640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:54.011641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:54.011724 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:54.012315 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:54.012317 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:54.012317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:54.012558 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:54.013695 (+ 1137us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:54.013854 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 15:22:54.014012 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 15:22:54.014696 17433 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.211.252" port: 36881 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:22:54.014182 17215 master_runner.cc:186] Error getting master registration for 127.3.211.253:39053: OK, instance_id { permanent_uuid: "b3810ae8a6254c23914f448659e0a8a8" instance_seqno: 1790349773803577 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 15:22:54.017539 17215 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 15:22:54.017642 17440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a55b8b55fb34b639002808e0d5c2c20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a55b8b55fb34b639002808e0d5c2c20" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 35715 } } peers { permanent_uuid: "b3810ae8a6254c23914f448659e0a8a8" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 39053 } } peers { permanent_uuid: "9cd022b0a54244f7927b3fc6d40377c0" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 36881 } } }
I20260925 15:22:54.017762 17440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a55b8b55fb34b639002808e0d5c2c20 [sys.catalog]: This master's current role is: FOLLOWER
Sep 25 15:22:54 dist-test-slave-4bk3 krb5kdc[17202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349773, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.252@KRBTEST.COM for kudu/127.3.211.252@KRBTEST.COM
Sep 25 15:22:54 dist-test-slave-4bk3 krb5kdc[17202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.211.253@KRBTEST.COM
I20260925 15:22:54.023375 17438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:54.014994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.252:36881 (local address 127.0.0.1:44260)
0925 15:22:54.015097 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:54.015110 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:54.015156 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:54.015336 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:54.015336 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:54.015538 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:22:54.015579 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:54.015582 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:54.016519 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:54.016520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:54.017240 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:54.017243 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:54.017744 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:54.017745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:54.017806 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:54.018246 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:22:54.018251 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:22:54.019907 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:22:54.022116 (+ 2209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:54.022118 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:54.022118 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:54.022353 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:54.022597 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:54.022598 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:54.022598 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:54.022665 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:54.023010 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:22:54.023010 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:22:54.023051 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 15:22:54.023122 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 15:22:54.023243 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260925 15:22:54.023489 17437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:54.015099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44260 (local address 127.3.211.252:36881)
0925 15:22:54.015142 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:54.015144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:54.015153 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:22:54.015232 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:54.015233 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:54.015248 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:54.015293 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:54.015658 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:54.016430 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:54.017336 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:54.017658 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:54.017731 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:54.020009 (+ 2278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:54.020010 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:54.020011 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:54.020019 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:54.022006 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:54.022447 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:54.022448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:54.022449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:54.022513 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:54.022732 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:54.022733 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:54.022733 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:54.022942 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:54.023002 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:54.023153 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 15:22:54.023314 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":24}
I20260925 15:22:54.024286 17453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:54.013460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33664 (local address 127.3.211.253:39053)
0925 15:22:54.013732 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:54.013735 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:54.013744 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:22:54.013769 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:54.013770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:54.013783 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:54.014498 (+ 715us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:54.014566 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:54.015326 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:54.016286 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:54.016656 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:54.017011 (+ 355us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:54.020214 (+ 3203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:54.020216 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:54.020217 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:54.020227 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:54.022540 (+ 2313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:54.022952 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:54.022954 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:54.022954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:54.023016 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:54.023226 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:54.023227 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:54.023227 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:54.023366 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:54.023931 (+ 565us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:54.023975 (+ 44us) server_negotiation.cc:299] Negotiation successful
0925 15:22:54.024104 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":156,"threads_started":1}
I20260925 15:22:54.025269 17433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cd022b0a54244f7927b3fc6d40377c0: Bootstrap starting.
I20260925 15:22:54.026086 17433 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9cd022b0a54244f7927b3fc6d40377c0: Neither blocks nor log segments found. Creating new log.
I20260925 15:22:54.026404 17433 log.cc:824] T 00000000000000000000000000000000 P 9cd022b0a54244f7927b3fc6d40377c0: Log is configured to *not* fsync() on all Append() calls
I20260925 15:22:54.027263 17433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cd022b0a54244f7927b3fc6d40377c0: No bootstrap required, opened a new log
I20260925 15:22:54.027797 3919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:22:54.029219 17433 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cd022b0a54244f7927b3fc6d40377c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a55b8b55fb34b639002808e0d5c2c20" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 35715 } } peers { permanent_uuid: "b3810ae8a6254c23914f448659e0a8a8" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 39053 } } peers { permanent_uuid: "9cd022b0a54244f7927b3fc6d40377c0" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 36881 } }
I20260925 15:22:54.029410 17433 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9cd022b0a54244f7927b3fc6d40377c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:22:54.029458 17433 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cd022b0a54244f7927b3fc6d40377c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cd022b0a54244f7927b3fc6d40377c0, State: Initialized, Role: FOLLOWER
I20260925 15:22:54.029605 17433 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cd022b0a54244f7927b3fc6d40377c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a55b8b55fb34b639002808e0d5c2c20" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 35715 } } peers { permanent_uuid: "b3810ae8a6254c23914f448659e0a8a8" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 39053 } } peers { permanent_uuid: "9cd022b0a54244f7927b3fc6d40377c0" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 36881 } }
I20260925 15:22:54.029942 17433 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cd022b0a54244f7927b3fc6d40377c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:22:54.030678 17461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cd022b0a54244f7927b3fc6d40377c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a55b8b55fb34b639002808e0d5c2c20" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 35715 } } peers { permanent_uuid: "b3810ae8a6254c23914f448659e0a8a8" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 39053 } } peers { permanent_uuid: "9cd022b0a54244f7927b3fc6d40377c0" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 36881 } } }
I20260925 15:22:54.030771 17461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cd022b0a54244f7927b3fc6d40377c0 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:22:54.033850 17365 master_runner.cc:186] Error getting master registration for 127.3.211.253:39053: OK, instance_id { permanent_uuid: "b3810ae8a6254c23914f448659e0a8a8" instance_seqno: 1790349773803577 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 15:22:54.036762 17365 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 15:22:54.039642 17473 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9cd022b0a54244f7927b3fc6d40377c0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:22:54.039700 17473 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:22:54.041005 17453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:54.029351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33672 (local address 127.3.211.253:39053)
0925 15:22:54.029770 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:54.029772 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:54.030043 (+ 271us) server_negotiation.cc:407] Connection header received
0925 15:22:54.030173 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:54.030174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:54.030189 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:54.030219 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:54.032019 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:54.032847 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:54.034028 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:54.034388 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:54.034432 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:54.036514 (+ 2082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:54.036515 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:54.036516 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:54.036524 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:54.038670 (+ 2146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:54.040076 (+ 1406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:54.040078 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:54.040078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:54.040143 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:54.040390 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:54.040390 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:54.040391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:54.040522 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:54.040614 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:54.040817 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 15:22:54.040906 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327}
I20260925 15:22:54.041218 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:54.029356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44268 (local address 127.3.211.254:35715)
0925 15:22:54.029400 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:54.029402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:54.029974 (+ 572us) server_negotiation.cc:407] Connection header received
0925 15:22:54.030078 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:54.030079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:54.030090 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:54.030120 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:54.031670 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:54.032472 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:54.033907 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:54.034243 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:54.034288 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:54.036514 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:54.036515 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:54.036516 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:54.036524 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:54.039346 (+ 2822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:54.039872 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:54.039873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:54.039874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:54.039957 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:54.040206 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:54.040207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:54.040207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:54.040401 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:54.040518 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:54.041002 (+ 484us) server_negotiation.cc:299] Negotiation successful
0925 15:22:54.041088 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 15:22:54.042953 17358 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.211.252" port: 36881 } attempt: 3
I20260925 15:22:54.044261 17437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:54.029716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44262 (local address 127.3.211.252:36881)
0925 15:22:54.029750 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:54.029751 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:54.030416 (+ 665us) server_negotiation.cc:407] Connection header received
0925 15:22:54.030511 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:54.030512 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:54.030522 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:54.030561 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:54.031462 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:54.032263 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:54.037854 (+ 5591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:54.038172 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:54.038212 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:54.040316 (+ 2104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:54.040317 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:54.040318 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:54.040326 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:54.042290 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:54.042675 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:54.042677 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:54.042677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:54.042743 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:54.043400 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:54.043401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:54.043401 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:54.043584 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:54.043693 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:54.044081 (+ 388us) server_negotiation.cc:299] Negotiation successful
0925 15:22:54.044165 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Sep 25 15:22:54 dist-test-slave-4bk3 krb5kdc[17202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349773, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.253@KRBTEST.COM for kudu/127.3.211.252@KRBTEST.COM
I20260925 15:22:54.052448 17435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:54.043250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44270 (local address 127.3.211.252:36881)
0925 15:22:54.043298 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:54.043300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:54.043310 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:22:54.043378 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:54.043379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:54.043391 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:54.043447 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:54.043795 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:54.044651 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:54.045724 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:54.046062 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:54.046136 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:54.049111 (+ 2975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:54.049112 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:54.049112 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:54.049120 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:54.050955 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:54.051496 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:54.051497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:54.051498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:54.051571 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:54.051807 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:54.051808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:54.051808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:54.051959 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:54.052028 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:54.052220 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 15:22:54.052306 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 15:22:54.052460 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:54.043206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.252:36881 (local address 127.0.0.1:44270)
0925 15:22:54.043257 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:54.043269 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:54.043312 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:54.043497 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:54.043498 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:54.043680 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:22:54.043729 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:54.043731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:54.044766 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:54.044768 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:54.045602 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:54.045605 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:54.046166 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:54.046167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:54.046240 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:54.046791 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:22:54.046797 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:22:54.049002 (+ 2205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:22:54.051080 (+ 2078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:54.051082 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:54.051083 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:54.051386 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:54.051652 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:54.051653 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:54.051654 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:54.051730 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:54.052040 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:22:54.052041 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:22:54.052100 (+ 59us) client_negotiation.cc:769] Sending connection context
0925 15:22:54.052186 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 15:22:54.052309 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 15:22:54.055039 17358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3810ae8a6254c23914f448659e0a8a8: Bootstrap starting.
I20260925 15:22:54.055835 17358 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3810ae8a6254c23914f448659e0a8a8: Neither blocks nor log segments found. Creating new log.
I20260925 15:22:54.056142 17358 log.cc:824] T 00000000000000000000000000000000 P b3810ae8a6254c23914f448659e0a8a8: Log is configured to *not* fsync() on all Append() calls
I20260925 15:22:54.057094 17358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3810ae8a6254c23914f448659e0a8a8: No bootstrap required, opened a new log
I20260925 15:22:54.059571 17358 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3810ae8a6254c23914f448659e0a8a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a55b8b55fb34b639002808e0d5c2c20" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 35715 } } peers { permanent_uuid: "b3810ae8a6254c23914f448659e0a8a8" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 39053 } } peers { permanent_uuid: "9cd022b0a54244f7927b3fc6d40377c0" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 36881 } }
I20260925 15:22:54.059813 17358 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3810ae8a6254c23914f448659e0a8a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:22:54.059875 17358 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3810ae8a6254c23914f448659e0a8a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3810ae8a6254c23914f448659e0a8a8, State: Initialized, Role: FOLLOWER
I20260925 15:22:54.060040 17358 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3810ae8a6254c23914f448659e0a8a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a55b8b55fb34b639002808e0d5c2c20" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 35715 } } peers { permanent_uuid: "b3810ae8a6254c23914f448659e0a8a8" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 39053 } } peers { permanent_uuid: "9cd022b0a54244f7927b3fc6d40377c0" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 36881 } }
I20260925 15:22:54.060424 17476 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3810ae8a6254c23914f448659e0a8a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a55b8b55fb34b639002808e0d5c2c20" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 35715 } } peers { permanent_uuid: "b3810ae8a6254c23914f448659e0a8a8" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 39053 } } peers { permanent_uuid: "9cd022b0a54244f7927b3fc6d40377c0" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 36881 } } }
I20260925 15:22:54.060549 17476 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3810ae8a6254c23914f448659e0a8a8 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:22:54.060489 17358 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3810ae8a6254c23914f448659e0a8a8 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 15:22:54.063609 17487 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b3810ae8a6254c23914f448659e0a8a8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:22:54.063745 17487 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:22:54.065608 17290 master_runner.cc:263] No leader master found from master b3810ae8a6254c23914f448659e0a8a8
I20260925 15:22:54.065677 17290 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 15:22:54.065738 17476 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3810ae8a6254c23914f448659e0a8a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:22:54.065807 17476 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3810ae8a6254c23914f448659e0a8a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a55b8b55fb34b639002808e0d5c2c20" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 35715 } } peers { permanent_uuid: "b3810ae8a6254c23914f448659e0a8a8" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 39053 } } peers { permanent_uuid: "9cd022b0a54244f7927b3fc6d40377c0" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 36881 } }
I20260925 15:22:54.066214 17257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b3810ae8a6254c23914f448659e0a8a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a55b8b55fb34b639002808e0d5c2c20" is_pre_election: true
I20260925 15:22:54.066390 17257 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7a55b8b55fb34b639002808e0d5c2c20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3810ae8a6254c23914f448659e0a8a8 in term 0.
I20260925 15:22:54.066596 17307 leader_election.cc:302] T 00000000000000000000000000000000 P b3810ae8a6254c23914f448659e0a8a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a55b8b55fb34b639002808e0d5c2c20, b3810ae8a6254c23914f448659e0a8a8; no voters:
I20260925 15:22:54.066996 17407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b3810ae8a6254c23914f448659e0a8a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cd022b0a54244f7927b3fc6d40377c0" is_pre_election: true
I20260925 15:22:54.067142 17407 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9cd022b0a54244f7927b3fc6d40377c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3810ae8a6254c23914f448659e0a8a8 in term 0.
I20260925 15:22:54.067319 17476 leader_election.cc:288] T 00000000000000000000000000000000 P b3810ae8a6254c23914f448659e0a8a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a55b8b55fb34b639002808e0d5c2c20 (127.3.211.254:35715), 9cd022b0a54244f7927b3fc6d40377c0 (127.3.211.252:36881)
I20260925 15:22:54.067422 17476 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3810ae8a6254c23914f448659e0a8a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:22:54.067469 17476 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3810ae8a6254c23914f448659e0a8a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:22:54.067497 17476 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3810ae8a6254c23914f448659e0a8a8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:22:54.068576 17476 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3810ae8a6254c23914f448659e0a8a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a55b8b55fb34b639002808e0d5c2c20" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 35715 } } peers { permanent_uuid: "b3810ae8a6254c23914f448659e0a8a8" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 39053 } } peers { permanent_uuid: "9cd022b0a54244f7927b3fc6d40377c0" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 36881 } }
I20260925 15:22:54.068845 17257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b3810ae8a6254c23914f448659e0a8a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a55b8b55fb34b639002808e0d5c2c20"
I20260925 15:22:54.068933 17257 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a55b8b55fb34b639002808e0d5c2c20 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:22:54.069875 17257 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7a55b8b55fb34b639002808e0d5c2c20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3810ae8a6254c23914f448659e0a8a8 in term 1.
I20260925 15:22:54.070822 17307 leader_election.cc:302] T 00000000000000000000000000000000 P b3810ae8a6254c23914f448659e0a8a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7a55b8b55fb34b639002808e0d5c2c20, b3810ae8a6254c23914f448659e0a8a8; no voters:
I20260925 15:22:54.071246 17407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b3810ae8a6254c23914f448659e0a8a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cd022b0a54244f7927b3fc6d40377c0"
I20260925 15:22:54.071341 17407 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cd022b0a54244f7927b3fc6d40377c0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:22:54.072556 17407 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9cd022b0a54244f7927b3fc6d40377c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3810ae8a6254c23914f448659e0a8a8 in term 1.
I20260925 15:22:54.072956 17476 leader_election.cc:288] T 00000000000000000000000000000000 P b3810ae8a6254c23914f448659e0a8a8 [CANDIDATE]: Term 1 election: Requested vote from peers 7a55b8b55fb34b639002808e0d5c2c20 (127.3.211.254:35715), 9cd022b0a54244f7927b3fc6d40377c0 (127.3.211.252:36881)
I20260925 15:22:54.073053 17476 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3810ae8a6254c23914f448659e0a8a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:22:54.073102 17476 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3810ae8a6254c23914f448659e0a8a8 [term 1 LEADER]: Becoming Leader. State: Replica: b3810ae8a6254c23914f448659e0a8a8, State: Running, Role: LEADER
I20260925 15:22:54.073195 17476 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3810ae8a6254c23914f448659e0a8a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a55b8b55fb34b639002808e0d5c2c20" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 35715 } } peers { permanent_uuid: "b3810ae8a6254c23914f448659e0a8a8" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 39053 } } peers { permanent_uuid: "9cd022b0a54244f7927b3fc6d40377c0" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 36881 } }
I20260925 15:22:54.073699 17476 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3810ae8a6254c23914f448659e0a8a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3810ae8a6254c23914f448659e0a8a8. Latest consensus state: current_term: 1 leader_uuid: "b3810ae8a6254c23914f448659e0a8a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a55b8b55fb34b639002808e0d5c2c20" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 35715 } } peers { permanent_uuid: "b3810ae8a6254c23914f448659e0a8a8" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 39053 } } peers { permanent_uuid: "9cd022b0a54244f7927b3fc6d40377c0" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 36881 } } }
I20260925 15:22:54.073786 17476 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3810ae8a6254c23914f448659e0a8a8 [sys.catalog]: This master's current role is: LEADER
I20260925 15:22:54.077947 17491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:22:54.078191 17491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:22:54.081305 17407 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9cd022b0a54244f7927b3fc6d40377c0 [term 1 FOLLOWER]: Refusing update from remote peer b3810ae8a6254c23914f448659e0a8a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:22:54.081691 17489 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b3810ae8a6254c23914f448659e0a8a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cd022b0a54244f7927b3fc6d40377c0" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 36881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:22:54.083119 17461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cd022b0a54244f7927b3fc6d40377c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3810ae8a6254c23914f448659e0a8a8. Latest consensus state: current_term: 1 leader_uuid: "b3810ae8a6254c23914f448659e0a8a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a55b8b55fb34b639002808e0d5c2c20" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 35715 } } peers { permanent_uuid: "b3810ae8a6254c23914f448659e0a8a8" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 39053 } } peers { permanent_uuid: "9cd022b0a54244f7927b3fc6d40377c0" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 36881 } } }
I20260925 15:22:54.083235 17461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cd022b0a54244f7927b3fc6d40377c0 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:22:54.083711 17257 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7a55b8b55fb34b639002808e0d5c2c20 [term 1 FOLLOWER]: Refusing update from remote peer b3810ae8a6254c23914f448659e0a8a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:22:54.084012 17489 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b3810ae8a6254c23914f448659e0a8a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a55b8b55fb34b639002808e0d5c2c20" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 35715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:22:54.086158 17440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a55b8b55fb34b639002808e0d5c2c20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3810ae8a6254c23914f448659e0a8a8. Latest consensus state: current_term: 1 leader_uuid: "b3810ae8a6254c23914f448659e0a8a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a55b8b55fb34b639002808e0d5c2c20" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 35715 } } peers { permanent_uuid: "b3810ae8a6254c23914f448659e0a8a8" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 39053 } } peers { permanent_uuid: "9cd022b0a54244f7927b3fc6d40377c0" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 36881 } } }
I20260925 15:22:54.086279 17440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a55b8b55fb34b639002808e0d5c2c20 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:22:54.087364 17489 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3810ae8a6254c23914f448659e0a8a8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b3810ae8a6254c23914f448659e0a8a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a55b8b55fb34b639002808e0d5c2c20" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 35715 } } peers { permanent_uuid: "b3810ae8a6254c23914f448659e0a8a8" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 39053 } } peers { permanent_uuid: "9cd022b0a54244f7927b3fc6d40377c0" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 36881 } } }
I20260925 15:22:54.087471 17489 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3810ae8a6254c23914f448659e0a8a8 [sys.catalog]: This master's current role is: LEADER
I20260925 15:22:54.087879 17440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a55b8b55fb34b639002808e0d5c2c20 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b3810ae8a6254c23914f448659e0a8a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a55b8b55fb34b639002808e0d5c2c20" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 35715 } } peers { permanent_uuid: "b3810ae8a6254c23914f448659e0a8a8" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 39053 } } peers { permanent_uuid: "9cd022b0a54244f7927b3fc6d40377c0" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 36881 } } }
I20260925 15:22:54.087970 17440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a55b8b55fb34b639002808e0d5c2c20 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:22:54.088172 17492 mvcc.cc:205] Tried to move back new op lower bound from 7333272674634018816 to 7333272674604531712. Current Snapshot: MvccSnapshot[applied={T|T < 7333272674634018816}]
I20260925 15:22:54.088810 17495 mvcc.cc:205] Tried to move back new op lower bound from 7333272674634018816 to 7333272674604531712. Current Snapshot: MvccSnapshot[applied={T|T < 7333272674634018816}]
I20260925 15:22:54.089785 17476 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3810ae8a6254c23914f448659e0a8a8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b3810ae8a6254c23914f448659e0a8a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a55b8b55fb34b639002808e0d5c2c20" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 35715 } } peers { permanent_uuid: "b3810ae8a6254c23914f448659e0a8a8" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 39053 } } peers { permanent_uuid: "9cd022b0a54244f7927b3fc6d40377c0" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 36881 } } }
I20260925 15:22:54.089895 17476 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3810ae8a6254c23914f448659e0a8a8 [sys.catalog]: This master's current role is: LEADER
I20260925 15:22:54.091110 17461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cd022b0a54244f7927b3fc6d40377c0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b3810ae8a6254c23914f448659e0a8a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7a55b8b55fb34b639002808e0d5c2c20" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 35715 } } peers { permanent_uuid: "b3810ae8a6254c23914f448659e0a8a8" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 39053 } } peers { permanent_uuid: "9cd022b0a54244f7927b3fc6d40377c0" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 36881 } } }
I20260925 15:22:54.091229 17461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cd022b0a54244f7927b3fc6d40377c0 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:22:54.092453 17493 mvcc.cc:205] Tried to move back new op lower bound from 7333272674634018816 to 7333272674604531712. Current Snapshot: MvccSnapshot[applied={T|T < 7333272674634018816}]
I20260925 15:22:54.093047 17491 catalog_manager.cc:1347] Generated new cluster ID: 19faf50a823d44a6bd5fe77cb233fd32
I20260925 15:22:54.093119 17491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:22:54.125051 17491 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:22:54.125885 17491 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:22:54.138482 17491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3810ae8a6254c23914f448659e0a8a8: Generated new TSK 0
I20260925 15:22:54.139743 17491 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:22:55.013056 17451 catalog_manager.cc:1259] Loaded cluster ID: 19faf50a823d44a6bd5fe77cb233fd32
I20260925 15:22:55.013172 17451 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7a55b8b55fb34b639002808e0d5c2c20: loading cluster ID for follower catalog manager: success
I20260925 15:22:55.014168 17451 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7a55b8b55fb34b639002808e0d5c2c20: acquiring CA information for follower catalog manager: success
I20260925 15:22:55.014570 17451 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7a55b8b55fb34b639002808e0d5c2c20: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 15:22:55.040198 17473 catalog_manager.cc:1259] Loaded cluster ID: 19faf50a823d44a6bd5fe77cb233fd32
I20260925 15:22:55.040287 17473 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9cd022b0a54244f7927b3fc6d40377c0: loading cluster ID for follower catalog manager: success
I20260925 15:22:55.041356 17473 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9cd022b0a54244f7927b3fc6d40377c0: acquiring CA information for follower catalog manager: success
I20260925 15:22:55.041776 17473 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9cd022b0a54244f7927b3fc6d40377c0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 15:22:55.209180 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 17215
I20260925 15:22:55.214504 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 17290
I20260925 15:22:55.220047 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 17365
2026-09-25T15:22:55Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2927 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349610595785-3919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:22:55 dist-test-slave-4bk3 krb5kdc[17506](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:22:55 dist-test-slave-4bk3 krb5kdc[17506](info): set up 2 sockets
Sep 25 15:22:55 dist-test-slave-4bk3 krb5kdc[17506](info): commencing operation
krb5kdc: starting...
W20260925 15:22:56.288550 3919 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Sep 25 15:22:56 dist-test-slave-4bk3 krb5kdc[17506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:22:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:22:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.211.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
I20260925 15:22:56.411674 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:42053
--webserver_interface=127.3.211.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:44309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.211.254:42053
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:22:56.498062 17519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:22:56.498312 17519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:22:56.498339 17519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:22:56.499614 17519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:22:56.499662 17519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:22:56.499677 17519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:22:56.499691 17519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:22:56.499702 17519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:22:56.501392 17519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:44309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349610595785-3919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.211.254:42053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:42053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.211.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:22:56.501659 17519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:22:56.501853 17519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:22:56.507988 17527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:56.507994 17525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:22:56.508030 17519 server_base.cc:1034] running on GCE node
W20260925 15:22:56.507994 17524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:22:56.508672 17519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:22:56.509011 17519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:22:56.510205 17519 hybrid_clock.cc:648] HybridClock initialized: now 1790349776510190 us; error 33 us; skew 500 ppm
Sep 25 15:22:56 dist-test-slave-4bk3 krb5kdc[17506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349776, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:22:56.513226 17519 init.cc:377] Logged in from keytab as kudu/127.3.211.254@KRBTEST.COM (short username kudu)
I20260925 15:22:56.513751 17519 webserver.cc:460] Webserver started at http://127.3.211.254:43699/ using document root <none> and password file <none>
I20260925 15:22:56.514019 17519 fs_manager.cc:359] Metadata directory not provided
I20260925 15:22:56.514070 17519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:22:56.514222 17519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:22:56.515751 17519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349610595785-3919-0/minicluster-data/master-0/data/instance:
uuid: "0aab55a888c2427fb34f3d937c86dd3f"
format_stamp: "Formatted at 2026-09-25 15:22:56 on dist-test-slave-4bk3"
server_key: "cd47b120ae76d5892780eb9d9097f895"
server_key_iv: "feb4dc2039cccc972a1021bc0e4524c4"
server_key_version: "encryptionkey@0"
I20260925 15:22:56.516212 17519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349610595785-3919-0/minicluster-data/master-0/wal/instance:
uuid: "0aab55a888c2427fb34f3d937c86dd3f"
format_stamp: "Formatted at 2026-09-25 15:22:56 on dist-test-slave-4bk3"
server_key: "cd47b120ae76d5892780eb9d9097f895"
server_key_iv: "feb4dc2039cccc972a1021bc0e4524c4"
server_key_version: "encryptionkey@0"
I20260925 15:22:56.517838 17519 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:22:56.518836 17534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:56.519048 17519 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:22:56.519153 17519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349610595785-3919-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349610595785-3919-0/minicluster-data/master-0/wal
uuid: "0aab55a888c2427fb34f3d937c86dd3f"
format_stamp: "Formatted at 2026-09-25 15:22:56 on dist-test-slave-4bk3"
server_key: "cd47b120ae76d5892780eb9d9097f895"
server_key_iv: "feb4dc2039cccc972a1021bc0e4524c4"
server_key_version: "encryptionkey@0"
I20260925 15:22:56.519210 17519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349610595785-3919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349610595785-3919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349610595785-3919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:22:56.531376 17519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:22:56.534569 17519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:22:56.534708 17519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:22:56.539561 17519 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.254:42053
I20260925 15:22:56.539629 17586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.254:42053 every 8 connection(s)
I20260925 15:22:56.539999 17519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
I20260925 15:22:56.541049 17587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:22:56.543530 17587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0aab55a888c2427fb34f3d937c86dd3f: Bootstrap starting.
I20260925 15:22:56.544181 17587 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0aab55a888c2427fb34f3d937c86dd3f: Neither blocks nor log segments found. Creating new log.
I20260925 15:22:56.544551 17587 log.cc:824] T 00000000000000000000000000000000 P 0aab55a888c2427fb34f3d937c86dd3f: Log is configured to *not* fsync() on all Append() calls
I20260925 15:22:56.545362 17587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0aab55a888c2427fb34f3d937c86dd3f: No bootstrap required, opened a new log
I20260925 15:22:56.547147 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 17519
I20260925 15:22:56.547245 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790349610595785-3919-0/minicluster-data/master-0/wal/instance
I20260925 15:22:56.547117 17587 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0aab55a888c2427fb34f3d937c86dd3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aab55a888c2427fb34f3d937c86dd3f" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 42053 } }
I20260925 15:22:56.547310 17587 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0aab55a888c2427fb34f3d937c86dd3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:22:56.547353 17587 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0aab55a888c2427fb34f3d937c86dd3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aab55a888c2427fb34f3d937c86dd3f, State: Initialized, Role: FOLLOWER
I20260925 15:22:56.547443 3919 external_mini_cluster.cc:1428] Setting key e76d9b0a845cffa30daac1b7babdd2bf
I20260925 15:22:56.547580 17587 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0aab55a888c2427fb34f3d937c86dd3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aab55a888c2427fb34f3d937c86dd3f" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 42053 } }
I20260925 15:22:56.547683 17587 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0aab55a888c2427fb34f3d937c86dd3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:22:56.547709 17587 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0aab55a888c2427fb34f3d937c86dd3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:22:56.547741 17587 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0aab55a888c2427fb34f3d937c86dd3f [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:22:56.548964 17587 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0aab55a888c2427fb34f3d937c86dd3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aab55a888c2427fb34f3d937c86dd3f" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 42053 } }
I20260925 15:22:56.549072 17587 leader_election.cc:302] T 00000000000000000000000000000000 P 0aab55a888c2427fb34f3d937c86dd3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0aab55a888c2427fb34f3d937c86dd3f; no voters:
I20260925 15:22:56.549297 17587 leader_election.cc:288] T 00000000000000000000000000000000 P 0aab55a888c2427fb34f3d937c86dd3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:22:56.549326 17592 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0aab55a888c2427fb34f3d937c86dd3f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:22:56.549430 17592 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0aab55a888c2427fb34f3d937c86dd3f [term 1 LEADER]: Becoming Leader. State: Replica: 0aab55a888c2427fb34f3d937c86dd3f, State: Running, Role: LEADER
I20260925 15:22:56.549518 17592 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0aab55a888c2427fb34f3d937c86dd3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aab55a888c2427fb34f3d937c86dd3f" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 42053 } }
I20260925 15:22:56.549571 17587 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0aab55a888c2427fb34f3d937c86dd3f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:22:56.549899 17594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0aab55a888c2427fb34f3d937c86dd3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0aab55a888c2427fb34f3d937c86dd3f. Latest consensus state: current_term: 1 leader_uuid: "0aab55a888c2427fb34f3d937c86dd3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aab55a888c2427fb34f3d937c86dd3f" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 42053 } } }
I20260925 15:22:56.550160 17594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0aab55a888c2427fb34f3d937c86dd3f [sys.catalog]: This master's current role is: LEADER
I20260925 15:22:56.550087 17593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0aab55a888c2427fb34f3d937c86dd3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0aab55a888c2427fb34f3d937c86dd3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aab55a888c2427fb34f3d937c86dd3f" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 42053 } } }
I20260925 15:22:56.550413 17593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0aab55a888c2427fb34f3d937c86dd3f [sys.catalog]: This master's current role is: LEADER
I20260925 15:22:56.550673 17601 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:22:56.551328 17601 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:22:56.553313 17601 catalog_manager.cc:1347] Generated new cluster ID: 88bd0b4f0767426eb849231e235c1a96
I20260925 15:22:56.553363 17601 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 15:22:56 dist-test-slave-4bk3 krb5kdc[17506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:22:56.563611 17591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:56.548104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57046 (local address 127.3.211.254:42053)
0925 15:22:56.548320 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:56.548353 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:56.548396 (+ 43us) server_negotiation.cc:407] Connection header received
0925 15:22:56.548580 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:56.548589 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:56.548664 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:56.548828 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:56.549767 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:56.551331 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:56.552539 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:56.553075 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:56.553212 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:56.559606 (+ 6394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:56.559611 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:56.559612 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:56.559629 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:56.561766 (+ 2137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:56.562222 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:56.562225 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:56.562225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:56.562331 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:56.562555 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:56.562556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:56.562556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:56.562872 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:56.563022 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:56.563135 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 15:22:56.563269 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":109,"threads_started":1}
I20260925 15:22:56.564538 3919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:22:56.566743 17601 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:22:56.567237 17601 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:22:56.573264 17601 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0aab55a888c2427fb34f3d937c86dd3f: Generated new TSK 0
I20260925 15:22:56.573482 17601 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:22:56.618866 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 17519
2026-09-25T15:22:56Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1396 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:22:56 dist-test-slave-4bk3 krb5kdc[17614](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:22:56 dist-test-slave-4bk3 krb5kdc[17614](info): set up 2 sockets
Sep 25 15:22:56 dist-test-slave-4bk3 krb5kdc[17614](info): commencing operation
krb5kdc: starting...
W20260925 15:22:57.701164 3919 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Sep 25 15:22:57 dist-test-slave-4bk3 krb5kdc[17614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:22:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:22:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.211.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
I20260925 15:22:57.839162 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:34373
--webserver_interface=127.3.211.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:46329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.211.254:34373
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:22:57.925098 17627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:22:57.925338 17627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:22:57.925364 17627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:22:57.926564 17627 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:22:57.926615 17627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:22:57.926630 17627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:22:57.926643 17627 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:22:57.926656 17627 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:22:57.928205 17627 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:46329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.211.254:34373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:34373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.211.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:22:57.928532 17627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:22:57.928737 17627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:22:57.934280 17633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:57.934297 17632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:57.934435 17635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:22:57.934297 17627 server_base.cc:1034] running on GCE node
I20260925 15:22:57.934736 17627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:22:57.935045 17627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:22:57.936273 17627 hybrid_clock.cc:648] HybridClock initialized: now 1790349777936256 us; error 50 us; skew 500 ppm
Sep 25 15:22:57 dist-test-slave-4bk3 krb5kdc[17614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349777, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:22:57.945837 17627 init.cc:377] Logged in from keytab as kudu/127.3.211.254@KRBTEST.COM (short username kudu)
I20260925 15:22:57.946362 17627 webserver.cc:460] Webserver started at http://127.3.211.254:40273/ using document root <none> and password file <none>
I20260925 15:22:57.946632 17627 fs_manager.cc:359] Metadata directory not provided
I20260925 15:22:57.946709 17627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:22:57.946867 17627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:22:57.948051 17627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/master-0/data/instance:
uuid: "d18c834f3f6e4638a02dcc3eed10aa68"
format_stamp: "Formatted at 2026-09-25 15:22:57 on dist-test-slave-4bk3"
server_key: "74c164be27e4b8b724c41986d65353ee"
server_key_iv: "726dbd089ad157cbb45e55c653fa7e52"
server_key_version: "encryptionkey@0"
I20260925 15:22:57.948531 17627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/master-0/wal/instance:
uuid: "d18c834f3f6e4638a02dcc3eed10aa68"
format_stamp: "Formatted at 2026-09-25 15:22:57 on dist-test-slave-4bk3"
server_key: "74c164be27e4b8b724c41986d65353ee"
server_key_iv: "726dbd089ad157cbb45e55c653fa7e52"
server_key_version: "encryptionkey@0"
I20260925 15:22:57.950042 17627 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:22:57.951179 17642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:57.951470 17627 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260925 15:22:57.951570 17627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/master-0/wal
uuid: "d18c834f3f6e4638a02dcc3eed10aa68"
format_stamp: "Formatted at 2026-09-25 15:22:57 on dist-test-slave-4bk3"
server_key: "74c164be27e4b8b724c41986d65353ee"
server_key_iv: "726dbd089ad157cbb45e55c653fa7e52"
server_key_version: "encryptionkey@0"
I20260925 15:22:57.951632 17627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:22:57.967800 17627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:22:57.970127 17627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:22:57.970249 17627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:22:57.974512 17627 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.254:34373
I20260925 15:22:57.974571 17694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.254:34373 every 8 connection(s)
I20260925 15:22:57.974962 17627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
I20260925 15:22:57.975957 17695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:22:57.978377 17695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68: Bootstrap starting.
I20260925 15:22:57.979099 17695 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68: Neither blocks nor log segments found. Creating new log.
I20260925 15:22:57.979398 17695 log.cc:824] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68: Log is configured to *not* fsync() on all Append() calls
I20260925 15:22:57.980224 17695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68: No bootstrap required, opened a new log
I20260925 15:22:57.982108 17695 raft_consensus.cc:348] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d18c834f3f6e4638a02dcc3eed10aa68" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 34373 } }
I20260925 15:22:57.982318 17695 raft_consensus.cc:374] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:22:57.982375 17695 raft_consensus.cc:729] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d18c834f3f6e4638a02dcc3eed10aa68, State: Initialized, Role: FOLLOWER
I20260925 15:22:57.982592 17695 consensus_queue.cc:260] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d18c834f3f6e4638a02dcc3eed10aa68" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 34373 } }
I20260925 15:22:57.982707 17695 raft_consensus.cc:388] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:22:57.982755 17695 raft_consensus.cc:482] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:22:57.982811 17695 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:22:57.983788 17695 raft_consensus.cc:504] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d18c834f3f6e4638a02dcc3eed10aa68" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 34373 } }
I20260925 15:22:57.983930 17695 leader_election.cc:302] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d18c834f3f6e4638a02dcc3eed10aa68; no voters:
I20260925 15:22:57.984158 17695 leader_election.cc:288] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:22:57.984206 17698 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:22:57.984428 17698 raft_consensus.cc:686] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68 [term 1 LEADER]: Becoming Leader. State: Replica: d18c834f3f6e4638a02dcc3eed10aa68, State: Running, Role: LEADER
I20260925 15:22:57.984498 17695 sys_catalog.cc:564] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:22:57.984624 17698 consensus_queue.cc:237] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d18c834f3f6e4638a02dcc3eed10aa68" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 34373 } }
I20260925 15:22:57.985003 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 17627
I20260925 15:22:57.985065 17699 sys_catalog.cc:455] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d18c834f3f6e4638a02dcc3eed10aa68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d18c834f3f6e4638a02dcc3eed10aa68" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 34373 } } }
I20260925 15:22:57.985126 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/master-0/wal/instance
I20260925 15:22:57.985149 17699 sys_catalog.cc:458] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68 [sys.catalog]: This master's current role is: LEADER
I20260925 15:22:57.985388 3919 external_mini_cluster.cc:1428] Setting key 5eeb4e940dce929d0eee33acfc7979c4
I20260925 15:22:57.985599 17700 sys_catalog.cc:455] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d18c834f3f6e4638a02dcc3eed10aa68. Latest consensus state: current_term: 1 leader_uuid: "d18c834f3f6e4638a02dcc3eed10aa68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d18c834f3f6e4638a02dcc3eed10aa68" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 34373 } } }
I20260925 15:22:57.985687 17700 sys_catalog.cc:458] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68 [sys.catalog]: This master's current role is: LEADER
I20260925 15:22:57.985983 17705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:22:57.986526 17705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:22:57.988098 17705 catalog_manager.cc:1347] Generated new cluster ID: 331dfeb1e768466396a7dee70f75b8e6
I20260925 15:22:57.988155 17705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 15:22:57 dist-test-slave-4bk3 krb5kdc[17614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:22:57.999117 17716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:57.986265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39442 (local address 127.3.211.254:34373)
0925 15:22:57.986445 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:57.986455 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:57.986486 (+ 31us) server_negotiation.cc:407] Connection header received
0925 15:22:57.986667 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:57.986676 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:57.986736 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:57.986909 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:57.987833 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:57.989260 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:57.990519 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:57.991178 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:57.991281 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:57.994423 (+ 3142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:57.994430 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:57.994430 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:57.994448 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:57.997328 (+ 2880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:57.997811 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:57.997815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:57.997815 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:57.997953 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:57.998211 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:57.998212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:57.998213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:57.998510 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:57.998689 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:57.998764 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 15:22:57.998922 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":89,"threads_started":1}
I20260925 15:22:57.999225 17705 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:22:57.999846 17705 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:22:58.003118 17705 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68: Generated new TSK 0
I20260925 15:22:58.003310 17705 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.211.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab.
Entry for principal kudu/127.3.211.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab.
I20260925 15:22:58.035250 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.193:0
--local_ip_for_outbound_sockets=127.3.211.193
--webserver_interface=127.3.211.193
--webserver_port=0
--tserver_master_addrs=127.3.211.254:34373
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab
--principal=kudu/127.3.211.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:46329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:22:58.122574 17721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:22:58.122807 17721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:22:58.122834 17721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:22:58.124089 17721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:22:58.124140 17721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:22:58.124169 17721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.193
I20260925 15:22:58.125738 17721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:46329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab
--principal=kudu/127.3.211.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.211.193
--webserver_port=0
--tserver_master_addrs=127.3.211.254:34373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.193
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:22:58.126020 17721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:22:58.126284 17721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:22:58.132551 17729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:58.132550 17727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:58.132562 17726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:22:58.132969 17721 server_base.cc:1034] running on GCE node
I20260925 15:22:58.133193 17721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:22:58.133522 17721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:22:58.134725 17721 hybrid_clock.cc:648] HybridClock initialized: now 1790349778134711 us; error 28 us; skew 500 ppm
Sep 25 15:22:58 dist-test-slave-4bk3 krb5kdc[17614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349778, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:22:58.138183 17721 init.cc:377] Logged in from keytab as kudu/127.3.211.193@KRBTEST.COM (short username kudu)
I20260925 15:22:58.138772 17721 webserver.cc:460] Webserver started at http://127.3.211.193:36791/ using document root <none> and password file <none>
I20260925 15:22:58.139039 17721 fs_manager.cc:359] Metadata directory not provided
I20260925 15:22:58.139117 17721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:22:58.139281 17721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:22:58.140441 17721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-0/data/instance:
uuid: "f06bc4bc858c491b893baed833d7e1cd"
format_stamp: "Formatted at 2026-09-25 15:22:58 on dist-test-slave-4bk3"
server_key: "fd1d7454747e3eac10ae4b0f703fa59a"
server_key_iv: "d9b085bd2ead1a20cb2e76205e491cd0"
server_key_version: "encryptionkey@0"
I20260925 15:22:58.140817 17721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-0/wal/instance:
uuid: "f06bc4bc858c491b893baed833d7e1cd"
format_stamp: "Formatted at 2026-09-25 15:22:58 on dist-test-slave-4bk3"
server_key: "fd1d7454747e3eac10ae4b0f703fa59a"
server_key_iv: "d9b085bd2ead1a20cb2e76205e491cd0"
server_key_version: "encryptionkey@0"
I20260925 15:22:58.142232 17721 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:22:58.143092 17736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:58.143326 17721 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:22:58.143450 17721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-0/wal
uuid: "f06bc4bc858c491b893baed833d7e1cd"
format_stamp: "Formatted at 2026-09-25 15:22:58 on dist-test-slave-4bk3"
server_key: "fd1d7454747e3eac10ae4b0f703fa59a"
server_key_iv: "d9b085bd2ead1a20cb2e76205e491cd0"
server_key_version: "encryptionkey@0"
I20260925 15:22:58.143543 17721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:22:58.152570 17721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:22:58.155270 17721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:22:58.155454 17721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:22:58.155716 17721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:22:58.156102 17721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:22:58.156163 17721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:58.156200 17721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:22:58.156251 17721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:58.162642 17721 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.193:34875
I20260925 15:22:58.162750 17849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.193:34875 every 8 connection(s)
I20260925 15:22:58.163101 17721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
Sep 25 15:22:58 dist-test-slave-4bk3 krb5kdc[17614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349778, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.193@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:22:58.170854 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 17721
I20260925 15:22:58.170958 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-0/wal/instance
I20260925 15:22:58.171219 3919 external_mini_cluster.cc:1428] Setting key d7375e7e5e5414863a8461255a158fb0
I20260925 15:22:58.173663 17716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:58.163968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:48953 (local address 127.3.211.254:34373)
0925 15:22:58.164044 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:58.164046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:58.164258 (+ 212us) server_negotiation.cc:407] Connection header received
0925 15:22:58.164771 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:58.164773 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:58.164787 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:58.164858 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:58.165866 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:58.166401 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:58.167508 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:58.167717 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:58.167790 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:58.170256 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:58.170258 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:58.170258 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:58.170266 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:58.172139 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:58.172631 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:58.172632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:58.172633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:58.172699 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:58.172955 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:58.172956 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:58.172956 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:58.173110 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:58.173175 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:58.173448 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 15:22:58.173567 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 15:22:58.173884 17852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:58.163936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.193:48953)
0925 15:22:58.164095 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:58.164140 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:58.164667 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:58.164913 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:58.164916 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:58.165513 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:22:58.165735 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:58.165741 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:58.166538 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:58.166539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:58.167379 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:58.167381 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:58.167849 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:58.167850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:58.167980 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:58.168619 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:22:58.168628 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:22:58.170151 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:22:58.172219 (+ 2068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:58.172227 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:58.172229 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:58.172557 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:58.172768 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:58.172768 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:58.172768 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:58.172879 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:58.173168 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:22:58.173173 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:22:58.173299 (+ 126us) client_negotiation.cc:769] Sending connection context
0925 15:22:58.173406 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 15:22:58.173579 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":87,"threads_started":1}
I20260925 15:22:58.174528 17850 heartbeater.cc:347] Connected to a master server at 127.3.211.254:34373
I20260925 15:22:58.174629 17850 heartbeater.cc:464] Registering TS with master...
I20260925 15:22:58.174837 17850 heartbeater.cc:511] Master 127.3.211.254:34373 requested a full tablet report, sending...
I20260925 15:22:58.175472 17659 ts_manager.cc:194] Registered new tserver with Master: f06bc4bc858c491b893baed833d7e1cd (127.3.211.193:34875)
I20260925 15:22:58.176200 17659 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.193@KRBTEST.COM'} at 127.3.211.193:48953
WARNING: no policy specified for kudu/127.3.211.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab.
Entry for principal kudu/127.3.211.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab.
I20260925 15:22:58.199800 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.194:0
--local_ip_for_outbound_sockets=127.3.211.194
--webserver_interface=127.3.211.194
--webserver_port=0
--tserver_master_addrs=127.3.211.254:34373
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab
--principal=kudu/127.3.211.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:46329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:22:58.286971 17855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:22:58.287200 17855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:22:58.287228 17855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:22:58.288552 17855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:22:58.288607 17855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:22:58.288637 17855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.194
I20260925 15:22:58.290155 17855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:46329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab
--principal=kudu/127.3.211.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.211.194
--webserver_port=0
--tserver_master_addrs=127.3.211.254:34373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.194
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:22:58.290443 17855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:22:58.290694 17855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:22:58.295980 17860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:58.295948 17863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:58.295944 17861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:22:58.296394 17855 server_base.cc:1034] running on GCE node
I20260925 15:22:58.296657 17855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:22:58.296957 17855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:22:58.298197 17855 hybrid_clock.cc:648] HybridClock initialized: now 1790349778298206 us; error 124 us; skew 500 ppm
Sep 25 15:22:58 dist-test-slave-4bk3 krb5kdc[17614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349778, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:22:58.300925 17855 init.cc:377] Logged in from keytab as kudu/127.3.211.194@KRBTEST.COM (short username kudu)
I20260925 15:22:58.301399 17855 webserver.cc:460] Webserver started at http://127.3.211.194:36343/ using document root <none> and password file <none>
I20260925 15:22:58.301643 17855 fs_manager.cc:359] Metadata directory not provided
I20260925 15:22:58.301715 17855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:22:58.301875 17855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:22:58.302969 17855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-1/data/instance:
uuid: "0ebeb29ff8794c69802c8ac93d4041f5"
format_stamp: "Formatted at 2026-09-25 15:22:58 on dist-test-slave-4bk3"
server_key: "f3d329eae3527319e15752293c4cf17d"
server_key_iv: "24d7c8213763f7105ea359c64b904acf"
server_key_version: "encryptionkey@0"
I20260925 15:22:58.303345 17855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-1/wal/instance:
uuid: "0ebeb29ff8794c69802c8ac93d4041f5"
format_stamp: "Formatted at 2026-09-25 15:22:58 on dist-test-slave-4bk3"
server_key: "f3d329eae3527319e15752293c4cf17d"
server_key_iv: "24d7c8213763f7105ea359c64b904acf"
server_key_version: "encryptionkey@0"
I20260925 15:22:58.305012 17855 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:22:58.305998 17870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:58.306370 17855 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:22:58.306505 17855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-1/wal
uuid: "0ebeb29ff8794c69802c8ac93d4041f5"
format_stamp: "Formatted at 2026-09-25 15:22:58 on dist-test-slave-4bk3"
server_key: "f3d329eae3527319e15752293c4cf17d"
server_key_iv: "24d7c8213763f7105ea359c64b904acf"
server_key_version: "encryptionkey@0"
I20260925 15:22:58.306622 17855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:22:58.327207 17855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:22:58.329723 17855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:22:58.329900 17855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:22:58.330154 17855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:22:58.330531 17855 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:22:58.330588 17855 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:58.330647 17855 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:22:58.330674 17855 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:58.337433 17855 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.194:40125
I20260925 15:22:58.337494 17983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.194:40125 every 8 connection(s)
I20260925 15:22:58.337859 17855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
Sep 25 15:22:58 dist-test-slave-4bk3 krb5kdc[17614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349778, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.194@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:22:58.345569 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 17855
I20260925 15:22:58.345698 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-1/wal/instance
I20260925 15:22:58.345970 3919 external_mini_cluster.cc:1428] Setting key d9f903c0c9785933cb7d78031666db57
I20260925 15:22:58.348783 17716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:58.338633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:49233 (local address 127.3.211.254:34373)
0925 15:22:58.338675 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:58.338677 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:58.338882 (+ 205us) server_negotiation.cc:407] Connection header received
0925 15:22:58.339355 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:58.339355 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:58.339366 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:58.339430 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:58.340394 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:58.340915 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:58.341966 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:58.342178 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:58.342248 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:58.345020 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:58.345022 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:58.345022 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:58.345038 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:58.346930 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:58.347569 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:58.347570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:58.347571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:58.347644 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:58.347941 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:58.347942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:58.347942 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:58.348086 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:58.348175 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:58.348569 (+ 394us) server_negotiation.cc:299] Negotiation successful
0925 15:22:58.348675 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 15:22:58.349170 17986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:58.338573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.194:49233)
0925 15:22:58.338744 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:58.338787 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:58.339271 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:58.339494 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:58.339497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:58.340075 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:22:58.340280 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:58.340285 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:58.341023 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:58.341024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:58.341848 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:58.341850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:58.342267 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:58.342268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:58.342375 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:58.343232 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:22:58.343243 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:22:58.344897 (+ 1654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:22:58.347062 (+ 2165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:58.347073 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:58.347077 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:58.347463 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:58.347726 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:58.347727 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:58.347728 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:58.347859 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:58.348163 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:22:58.348170 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:22:58.348400 (+ 230us) client_negotiation.cc:769] Sending connection context
0925 15:22:58.348544 (+ 144us) client_negotiation.cc:240] Negotiation successful
0925 15:22:58.348744 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":91,"threads_started":1}
I20260925 15:22:58.349722 17984 heartbeater.cc:347] Connected to a master server at 127.3.211.254:34373
I20260925 15:22:58.349834 17984 heartbeater.cc:464] Registering TS with master...
I20260925 15:22:58.350049 17984 heartbeater.cc:511] Master 127.3.211.254:34373 requested a full tablet report, sending...
I20260925 15:22:58.350483 17659 ts_manager.cc:194] Registered new tserver with Master: 0ebeb29ff8794c69802c8ac93d4041f5 (127.3.211.194:40125)
I20260925 15:22:58.350925 17659 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.194@KRBTEST.COM'} at 127.3.211.194:49233
WARNING: no policy specified for kudu/127.3.211.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab.
Entry for principal kudu/127.3.211.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab.
I20260925 15:22:58.375144 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.195:0
--local_ip_for_outbound_sockets=127.3.211.195
--webserver_interface=127.3.211.195
--webserver_port=0
--tserver_master_addrs=127.3.211.254:34373
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab
--principal=kudu/127.3.211.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:46329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:22:58.466393 17989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:22:58.466646 17989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:22:58.466673 17989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:22:58.468314 17989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:22:58.468465 17989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:22:58.468508 17989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.195
I20260925 15:22:58.470193 17989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:46329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab
--principal=kudu/127.3.211.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.211.195
--webserver_port=0
--tserver_master_addrs=127.3.211.254:34373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.195
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:22:58.470472 17989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:22:58.470743 17989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:22:58.476974 17995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:58.476974 17997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:22:58.477028 17989 server_base.cc:1034] running on GCE node
W20260925 15:22:58.476974 17994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:22:58.477789 17989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:22:58.478210 17989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:22:58.479444 17989 hybrid_clock.cc:648] HybridClock initialized: now 1790349778479423 us; error 45 us; skew 500 ppm
Sep 25 15:22:58 dist-test-slave-4bk3 krb5kdc[17614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349778, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:22:58.482342 17989 init.cc:377] Logged in from keytab as kudu/127.3.211.195@KRBTEST.COM (short username kudu)
I20260925 15:22:58.482827 17989 webserver.cc:460] Webserver started at http://127.3.211.195:41357/ using document root <none> and password file <none>
I20260925 15:22:58.483062 17989 fs_manager.cc:359] Metadata directory not provided
I20260925 15:22:58.483117 17989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:22:58.483276 17989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:22:58.484508 17989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-2/data/instance:
uuid: "aa9b764a299e4504aab87753ddfab3c5"
format_stamp: "Formatted at 2026-09-25 15:22:58 on dist-test-slave-4bk3"
server_key: "73509fac2ace0aa24b5499535efcf550"
server_key_iv: "886bf768a689649c7627c90016fd9a0f"
server_key_version: "encryptionkey@0"
I20260925 15:22:58.484879 17989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-2/wal/instance:
uuid: "aa9b764a299e4504aab87753ddfab3c5"
format_stamp: "Formatted at 2026-09-25 15:22:58 on dist-test-slave-4bk3"
server_key: "73509fac2ace0aa24b5499535efcf550"
server_key_iv: "886bf768a689649c7627c90016fd9a0f"
server_key_version: "encryptionkey@0"
I20260925 15:22:58.486756 17989 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260925 15:22:58.487754 18004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:58.488085 17989 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:22:58.488171 17989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-2/wal
uuid: "aa9b764a299e4504aab87753ddfab3c5"
format_stamp: "Formatted at 2026-09-25 15:22:58 on dist-test-slave-4bk3"
server_key: "73509fac2ace0aa24b5499535efcf550"
server_key_iv: "886bf768a689649c7627c90016fd9a0f"
server_key_version: "encryptionkey@0"
I20260925 15:22:58.488270 17989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:22:58.501056 17989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:22:58.504171 17989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:22:58.504297 17989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:22:58.504643 17989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:22:58.504997 17989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:22:58.505059 17989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:58.505105 17989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:22:58.505132 17989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:58.511996 17989 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.195:45859
I20260925 15:22:58.512039 18117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.195:45859 every 8 connection(s)
I20260925 15:22:58.512454 17989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
Sep 25 15:22:58 dist-test-slave-4bk3 krb5kdc[17614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349778, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.195@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:22:58.522244 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 17989
I20260925 15:22:58.522365 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-2/wal/instance
I20260925 15:22:58.522574 3919 external_mini_cluster.cc:1428] Setting key 597ab58600e42088617eb37974d6df7a
I20260925 15:22:58.523550 18120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:58.513189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.195:37113)
0925 15:22:58.513353 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:58.513397 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:58.514328 (+ 931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:58.514616 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:58.514620 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:58.514913 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:22:58.515119 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:58.515124 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:58.516058 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:58.516059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:58.517032 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:58.517035 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:58.517507 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:58.517508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:58.517633 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:58.518224 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:22:58.518241 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:22:58.519787 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:22:58.521577 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:58.521586 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:58.521588 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:58.521847 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:58.522112 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:58.522113 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:58.522114 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:58.522267 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:58.522758 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:22:58.522763 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:22:58.522968 (+ 205us) client_negotiation.cc:769] Sending connection context
0925 15:22:58.523101 (+ 133us) client_negotiation.cc:240] Negotiation successful
0925 15:22:58.523276 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":92,"threads_started":1}
I20260925 15:22:58.523828 17716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:58.513992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:37113 (local address 127.3.211.254:34373)
0925 15:22:58.514070 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:58.514072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:58.514085 (+ 13us) server_negotiation.cc:407] Connection header received
0925 15:22:58.514462 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:58.514464 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:58.514475 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:58.514552 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:58.515309 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:58.515904 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:58.517170 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:58.517406 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:58.517486 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:58.519919 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:58.519921 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:58.519921 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:58.519929 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:58.521471 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:58.521964 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:58.521966 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:58.521966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:58.522030 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:58.522467 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:58.522469 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:58.522469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:58.522653 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:58.522759 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:58.523308 (+ 549us) server_negotiation.cc:299] Negotiation successful
0925 15:22:58.523402 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 15:22:58.524214 18118 heartbeater.cc:347] Connected to a master server at 127.3.211.254:34373
I20260925 15:22:58.524437 18118 heartbeater.cc:464] Registering TS with master...
I20260925 15:22:58.524698 18118 heartbeater.cc:511] Master 127.3.211.254:34373 requested a full tablet report, sending...
I20260925 15:22:58.525162 17659 ts_manager.cc:194] Registered new tserver with Master: aa9b764a299e4504aab87753ddfab3c5 (127.3.211.195:45859)
I20260925 15:22:58.525599 17659 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.195@KRBTEST.COM'} at 127.3.211.195:37113
I20260925 15:22:58.535432 3919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:22:58.544777 17716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:58.537357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39454 (local address 127.3.211.254:34373)
0925 15:22:58.537443 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:58.537445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:58.537565 (+ 120us) server_negotiation.cc:407] Connection header received
0925 15:22:58.537664 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:58.537665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:58.537677 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:58.537767 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:58.538443 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:58.539010 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:58.539878 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:58.540103 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:58.540191 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:58.541678 (+ 1487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:58.541681 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:58.541681 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:58.541690 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:58.543313 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:58.543761 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:58.543762 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:58.543763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:58.543833 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:58.544071 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:58.544072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:58.544072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:58.544250 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:58.544417 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:58.544556 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 15:22:58.544659 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 15:22:58.547367 17659 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39454:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:22:58.549206 17659 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:22:58.556581 18130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:58.552132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.194:40125 (local address 127.0.0.1:50224)
0925 15:22:58.552298 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:58.552306 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:58.552375 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:58.552766 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:58.552767 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:58.552769 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:22:58.552813 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:58.552814 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:58.553956 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:58.553957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:58.555464 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:58.555467 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:58.556370 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:58.556371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:58.556429 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:58.556433 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:22:58.556485 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 15:22:58.556517 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":71,"thread_start_us":98,"threads_started":1}
I20260925 15:22:58.556653 18134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:58.552404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50224 (local address 127.3.211.194:40125)
0925 15:22:58.552601 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:58.552605 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:58.552614 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:22:58.552653 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:58.552655 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:58.552670 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:58.552881 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:22:58.552952 (+ 71us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:58.553664 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:58.555543 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:58.556246 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:58.556378 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:58.556432 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:58.556502 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 15:22:58.556553 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":96,"threads_started":1}
I20260925 15:22:58.556734 18133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:58.551905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.195:45859 (local address 127.0.0.1:52242)
0925 15:22:58.552269 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:58.552280 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:58.552364 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:58.552673 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:58.552676 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:58.552678 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:22:58.552762 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:58.552765 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:58.554195 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:58.554196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:58.555367 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:58.555370 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:58.556428 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:58.556429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:58.556514 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:58.556518 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:22:58.556602 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 15:22:58.556648 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":42,"thread_start_us":66,"threads_started":1}
I20260925 15:22:58.556972 18131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:58.551997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43336 (local address 127.3.211.193:34875)
0925 15:22:58.552503 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:58.552508 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:58.552518 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:22:58.552567 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:58.552569 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:58.552585 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:58.552915 (+ 330us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:22:58.552991 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:58.554062 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:58.555574 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:58.556101 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:58.556202 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:58.556271 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:58.556637 (+ 366us) server_negotiation.cc:299] Negotiation successful
0925 15:22:58.556893 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":72,"threads_started":1}
I20260925 15:22:58.557098 18132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:58.552009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52242 (local address 127.3.211.195:45859)
0925 15:22:58.552217 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:58.552222 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:58.552432 (+ 210us) server_negotiation.cc:407] Connection header received
0925 15:22:58.552472 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:58.552474 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:58.552487 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:58.552586 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:22:58.552832 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:58.554073 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:58.555473 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:58.556225 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:58.556305 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:58.556392 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:58.556836 (+ 444us) server_negotiation.cc:299] Negotiation successful
0925 15:22:58.556897 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":100,"threads_started":1}
I20260925 15:22:58.556581 18129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:58.551904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:34875 (local address 127.0.0.1:43336)
0925 15:22:58.552078 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:58.552124 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:58.552195 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:58.552709 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:58.552710 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:58.552713 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:22:58.552770 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:58.552772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:58.554208 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:58.554209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:58.555464 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:58.555467 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:58.556202 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:58.556203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:58.556286 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:58.556301 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 15:22:58.556404 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 15:22:58.556471 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":21,"thread_start_us":103,"threads_started":1}
I20260925 15:22:58.557547 17918 tablet_service.cc:1467] Processing CreateTablet for tablet 7462f50533c24164ace1ee10757282e8 (DEFAULT_TABLE table=test-table [id=020c9e5f8f744490a58269d119ab4e84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:22:58.557929 17918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7462f50533c24164ace1ee10757282e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:22:58.558034 17784 tablet_service.cc:1467] Processing CreateTablet for tablet 7462f50533c24164ace1ee10757282e8 (DEFAULT_TABLE table=test-table [id=020c9e5f8f744490a58269d119ab4e84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:22:58.558344 17784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7462f50533c24164ace1ee10757282e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:22:58.560930 18135 tablet_bootstrap.cc:492] T 7462f50533c24164ace1ee10757282e8 P 0ebeb29ff8794c69802c8ac93d4041f5: Bootstrap starting.
I20260925 15:22:58.561196 18136 tablet_bootstrap.cc:492] T 7462f50533c24164ace1ee10757282e8 P f06bc4bc858c491b893baed833d7e1cd: Bootstrap starting.
I20260925 15:22:58.561658 18135 tablet_bootstrap.cc:654] T 7462f50533c24164ace1ee10757282e8 P 0ebeb29ff8794c69802c8ac93d4041f5: Neither blocks nor log segments found. Creating new log.
I20260925 15:22:58.561566 18052 tablet_service.cc:1467] Processing CreateTablet for tablet 7462f50533c24164ace1ee10757282e8 (DEFAULT_TABLE table=test-table [id=020c9e5f8f744490a58269d119ab4e84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:22:58.561793 18136 tablet_bootstrap.cc:654] T 7462f50533c24164ace1ee10757282e8 P f06bc4bc858c491b893baed833d7e1cd: Neither blocks nor log segments found. Creating new log.
I20260925 15:22:58.561932 18052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7462f50533c24164ace1ee10757282e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:22:58.562268 18135 log.cc:824] T 7462f50533c24164ace1ee10757282e8 P 0ebeb29ff8794c69802c8ac93d4041f5: Log is configured to *not* fsync() on all Append() calls
I20260925 15:22:58.562325 18136 log.cc:824] T 7462f50533c24164ace1ee10757282e8 P f06bc4bc858c491b893baed833d7e1cd: Log is configured to *not* fsync() on all Append() calls
I20260925 15:22:58.563053 18135 tablet_bootstrap.cc:492] T 7462f50533c24164ace1ee10757282e8 P 0ebeb29ff8794c69802c8ac93d4041f5: No bootstrap required, opened a new log
I20260925 15:22:58.563119 18136 tablet_bootstrap.cc:492] T 7462f50533c24164ace1ee10757282e8 P f06bc4bc858c491b893baed833d7e1cd: No bootstrap required, opened a new log
I20260925 15:22:58.563153 18135 ts_tablet_manager.cc:1397] T 7462f50533c24164ace1ee10757282e8 P 0ebeb29ff8794c69802c8ac93d4041f5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:22:58.563195 18136 ts_tablet_manager.cc:1397] T 7462f50533c24164ace1ee10757282e8 P f06bc4bc858c491b893baed833d7e1cd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:22:58.564924 18135 raft_consensus.cc:348] T 7462f50533c24164ace1ee10757282e8 P 0ebeb29ff8794c69802c8ac93d4041f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f06bc4bc858c491b893baed833d7e1cd" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34875 } } peers { permanent_uuid: "aa9b764a299e4504aab87753ddfab3c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 45859 } } peers { permanent_uuid: "0ebeb29ff8794c69802c8ac93d4041f5" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40125 } }
I20260925 15:22:58.565119 18135 raft_consensus.cc:374] T 7462f50533c24164ace1ee10757282e8 P 0ebeb29ff8794c69802c8ac93d4041f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:22:58.565166 18135 raft_consensus.cc:729] T 7462f50533c24164ace1ee10757282e8 P 0ebeb29ff8794c69802c8ac93d4041f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ebeb29ff8794c69802c8ac93d4041f5, State: Initialized, Role: FOLLOWER
I20260925 15:22:58.565297 18135 consensus_queue.cc:260] T 7462f50533c24164ace1ee10757282e8 P 0ebeb29ff8794c69802c8ac93d4041f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f06bc4bc858c491b893baed833d7e1cd" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34875 } } peers { permanent_uuid: "aa9b764a299e4504aab87753ddfab3c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 45859 } } peers { permanent_uuid: "0ebeb29ff8794c69802c8ac93d4041f5" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40125 } }
I20260925 15:22:58.565394 18136 raft_consensus.cc:348] T 7462f50533c24164ace1ee10757282e8 P f06bc4bc858c491b893baed833d7e1cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f06bc4bc858c491b893baed833d7e1cd" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34875 } } peers { permanent_uuid: "aa9b764a299e4504aab87753ddfab3c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 45859 } } peers { permanent_uuid: "0ebeb29ff8794c69802c8ac93d4041f5" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40125 } }
I20260925 15:22:58.565591 18136 raft_consensus.cc:374] T 7462f50533c24164ace1ee10757282e8 P f06bc4bc858c491b893baed833d7e1cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:22:58.565613 18135 ts_tablet_manager.cc:1428] T 7462f50533c24164ace1ee10757282e8 P 0ebeb29ff8794c69802c8ac93d4041f5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 15:22:58.565641 18136 raft_consensus.cc:729] T 7462f50533c24164ace1ee10757282e8 P f06bc4bc858c491b893baed833d7e1cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f06bc4bc858c491b893baed833d7e1cd, State: Initialized, Role: FOLLOWER
I20260925 15:22:58.565661 17984 heartbeater.cc:503] Master 127.3.211.254:34373 was elected leader, sending a full tablet report...
I20260925 15:22:58.565758 18136 consensus_queue.cc:260] T 7462f50533c24164ace1ee10757282e8 P f06bc4bc858c491b893baed833d7e1cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f06bc4bc858c491b893baed833d7e1cd" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34875 } } peers { permanent_uuid: "aa9b764a299e4504aab87753ddfab3c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 45859 } } peers { permanent_uuid: "0ebeb29ff8794c69802c8ac93d4041f5" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40125 } }
I20260925 15:22:58.565848 18139 tablet_bootstrap.cc:492] T 7462f50533c24164ace1ee10757282e8 P aa9b764a299e4504aab87753ddfab3c5: Bootstrap starting.
I20260925 15:22:58.566025 18136 ts_tablet_manager.cc:1428] T 7462f50533c24164ace1ee10757282e8 P f06bc4bc858c491b893baed833d7e1cd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:22:58.566155 17850 heartbeater.cc:503] Master 127.3.211.254:34373 was elected leader, sending a full tablet report...
I20260925 15:22:58.566828 18139 tablet_bootstrap.cc:654] T 7462f50533c24164ace1ee10757282e8 P aa9b764a299e4504aab87753ddfab3c5: Neither blocks nor log segments found. Creating new log.
I20260925 15:22:58.567358 18139 log.cc:824] T 7462f50533c24164ace1ee10757282e8 P aa9b764a299e4504aab87753ddfab3c5: Log is configured to *not* fsync() on all Append() calls
I20260925 15:22:58.568081 18139 tablet_bootstrap.cc:492] T 7462f50533c24164ace1ee10757282e8 P aa9b764a299e4504aab87753ddfab3c5: No bootstrap required, opened a new log
I20260925 15:22:58.568173 18139 ts_tablet_manager.cc:1397] T 7462f50533c24164ace1ee10757282e8 P aa9b764a299e4504aab87753ddfab3c5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:22:58.570109 18139 raft_consensus.cc:348] T 7462f50533c24164ace1ee10757282e8 P aa9b764a299e4504aab87753ddfab3c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f06bc4bc858c491b893baed833d7e1cd" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34875 } } peers { permanent_uuid: "aa9b764a299e4504aab87753ddfab3c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 45859 } } peers { permanent_uuid: "0ebeb29ff8794c69802c8ac93d4041f5" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40125 } }
I20260925 15:22:58.570279 18139 raft_consensus.cc:374] T 7462f50533c24164ace1ee10757282e8 P aa9b764a299e4504aab87753ddfab3c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:22:58.570302 18139 raft_consensus.cc:729] T 7462f50533c24164ace1ee10757282e8 P aa9b764a299e4504aab87753ddfab3c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa9b764a299e4504aab87753ddfab3c5, State: Initialized, Role: FOLLOWER
I20260925 15:22:58.570436 18139 consensus_queue.cc:260] T 7462f50533c24164ace1ee10757282e8 P aa9b764a299e4504aab87753ddfab3c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f06bc4bc858c491b893baed833d7e1cd" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34875 } } peers { permanent_uuid: "aa9b764a299e4504aab87753ddfab3c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 45859 } } peers { permanent_uuid: "0ebeb29ff8794c69802c8ac93d4041f5" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40125 } }
I20260925 15:22:58.570765 18139 ts_tablet_manager.cc:1428] T 7462f50533c24164ace1ee10757282e8 P aa9b764a299e4504aab87753ddfab3c5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:22:58.570845 18118 heartbeater.cc:503] Master 127.3.211.254:34373 was elected leader, sending a full tablet report...
W20260925 15:22:58.588531 17985 tablet.cc:2369] T 7462f50533c24164ace1ee10757282e8 P 0ebeb29ff8794c69802c8ac93d4041f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:22:58.590919 18143 raft_consensus.cc:482] T 7462f50533c24164ace1ee10757282e8 P aa9b764a299e4504aab87753ddfab3c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:22:58.591090 18143 raft_consensus.cc:504] T 7462f50533c24164ace1ee10757282e8 P aa9b764a299e4504aab87753ddfab3c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f06bc4bc858c491b893baed833d7e1cd" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34875 } } peers { permanent_uuid: "aa9b764a299e4504aab87753ddfab3c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 45859 } } peers { permanent_uuid: "0ebeb29ff8794c69802c8ac93d4041f5" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40125 } }
I20260925 15:22:58.591419 18143 leader_election.cc:288] T 7462f50533c24164ace1ee10757282e8 P aa9b764a299e4504aab87753ddfab3c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f06bc4bc858c491b893baed833d7e1cd (127.3.211.193:34875), 0ebeb29ff8794c69802c8ac93d4041f5 (127.3.211.194:40125)
I20260925 15:22:58.594810 18120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:58.591663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:34875 (local address 127.3.211.195:43661)
0925 15:22:58.591741 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:58.591755 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:58.591826 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:58.592056 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:58.592057 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:58.592059 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:22:58.592133 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:58.592136 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:58.592942 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:58.592943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:58.593802 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:58.593805 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:58.594481 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:58.594482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:58.594579 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:58.594584 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:22:58.594650 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 15:22:58.594704 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":61}
I20260925 15:22:58.594816 18131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:58.591845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:43661 (local address 127.3.211.193:34875)
0925 15:22:58.591915 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:58.591917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:58.591925 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:22:58.591963 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:58.591965 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:58.591975 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:58.592039 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:22:58.592274 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:58.592857 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:58.593921 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:58.594366 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:58.594427 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:58.594434 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:58.594670 (+ 236us) server_negotiation.cc:299] Negotiation successful
0925 15:22:58.594721 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 15:22:58.594970 18134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:58.591978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:38339 (local address 127.3.211.194:40125)
0925 15:22:58.592023 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:58.592025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:58.592073 (+ 48us) server_negotiation.cc:407] Connection header received
0925 15:22:58.592197 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:58.592199 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:58.592210 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:58.592274 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:22:58.592473 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:58.592950 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:58.594036 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:58.594554 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:58.594649 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:58.594659 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:58.594834 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 15:22:58.594884 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 15:22:58.594940 18144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:58.591783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.194:40125 (local address 127.3.211.195:38339)
0925 15:22:58.592010 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:58.592020 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:58.592117 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:58.592286 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:58.592287 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:58.592288 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:22:58.592387 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:58.592389 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:58.593035 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:58.593036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:58.593922 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:58.593924 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:58.594674 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:58.594675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:58.594742 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:58.594747 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:22:58.594807 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 15:22:58.594851 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":130,"threads_started":1}
I20260925 15:22:58.595291 17804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7462f50533c24164ace1ee10757282e8" candidate_uuid: "aa9b764a299e4504aab87753ddfab3c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f06bc4bc858c491b893baed833d7e1cd" is_pre_election: true
I20260925 15:22:58.595515 17804 raft_consensus.cc:2454] T 7462f50533c24164ace1ee10757282e8 P f06bc4bc858c491b893baed833d7e1cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa9b764a299e4504aab87753ddfab3c5 in term 0.
I20260925 15:22:58.595557 17938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7462f50533c24164ace1ee10757282e8" candidate_uuid: "aa9b764a299e4504aab87753ddfab3c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ebeb29ff8794c69802c8ac93d4041f5" is_pre_election: true
I20260925 15:22:58.595741 17938 raft_consensus.cc:2454] T 7462f50533c24164ace1ee10757282e8 P 0ebeb29ff8794c69802c8ac93d4041f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa9b764a299e4504aab87753ddfab3c5 in term 0.
I20260925 15:22:58.595779 18007 leader_election.cc:302] T 7462f50533c24164ace1ee10757282e8 P aa9b764a299e4504aab87753ddfab3c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa9b764a299e4504aab87753ddfab3c5, f06bc4bc858c491b893baed833d7e1cd; no voters:
I20260925 15:22:58.595920 18143 raft_consensus.cc:2781] T 7462f50533c24164ace1ee10757282e8 P aa9b764a299e4504aab87753ddfab3c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:22:58.595985 18143 raft_consensus.cc:482] T 7462f50533c24164ace1ee10757282e8 P aa9b764a299e4504aab87753ddfab3c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:22:58.596017 18143 raft_consensus.cc:3037] T 7462f50533c24164ace1ee10757282e8 P aa9b764a299e4504aab87753ddfab3c5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:22:58.596817 18143 raft_consensus.cc:504] T 7462f50533c24164ace1ee10757282e8 P aa9b764a299e4504aab87753ddfab3c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f06bc4bc858c491b893baed833d7e1cd" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34875 } } peers { permanent_uuid: "aa9b764a299e4504aab87753ddfab3c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 45859 } } peers { permanent_uuid: "0ebeb29ff8794c69802c8ac93d4041f5" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40125 } }
I20260925 15:22:58.596935 18143 leader_election.cc:288] T 7462f50533c24164ace1ee10757282e8 P aa9b764a299e4504aab87753ddfab3c5 [CANDIDATE]: Term 1 election: Requested vote from peers f06bc4bc858c491b893baed833d7e1cd (127.3.211.193:34875), 0ebeb29ff8794c69802c8ac93d4041f5 (127.3.211.194:40125)
I20260925 15:22:58.597177 17804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7462f50533c24164ace1ee10757282e8" candidate_uuid: "aa9b764a299e4504aab87753ddfab3c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f06bc4bc858c491b893baed833d7e1cd"
I20260925 15:22:58.597223 17938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7462f50533c24164ace1ee10757282e8" candidate_uuid: "aa9b764a299e4504aab87753ddfab3c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ebeb29ff8794c69802c8ac93d4041f5"
I20260925 15:22:58.597270 17804 raft_consensus.cc:3037] T 7462f50533c24164ace1ee10757282e8 P f06bc4bc858c491b893baed833d7e1cd [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:22:58.597298 17938 raft_consensus.cc:3037] T 7462f50533c24164ace1ee10757282e8 P 0ebeb29ff8794c69802c8ac93d4041f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:22:58.598230 17938 raft_consensus.cc:2454] T 7462f50533c24164ace1ee10757282e8 P 0ebeb29ff8794c69802c8ac93d4041f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa9b764a299e4504aab87753ddfab3c5 in term 1.
I20260925 15:22:58.598230 17804 raft_consensus.cc:2454] T 7462f50533c24164ace1ee10757282e8 P f06bc4bc858c491b893baed833d7e1cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa9b764a299e4504aab87753ddfab3c5 in term 1.
I20260925 15:22:58.598477 18007 leader_election.cc:302] T 7462f50533c24164ace1ee10757282e8 P aa9b764a299e4504aab87753ddfab3c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebeb29ff8794c69802c8ac93d4041f5, aa9b764a299e4504aab87753ddfab3c5; no voters:
I20260925 15:22:58.598587 18143 raft_consensus.cc:2781] T 7462f50533c24164ace1ee10757282e8 P aa9b764a299e4504aab87753ddfab3c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:22:58.598767 18143 raft_consensus.cc:686] T 7462f50533c24164ace1ee10757282e8 P aa9b764a299e4504aab87753ddfab3c5 [term 1 LEADER]: Becoming Leader. State: Replica: aa9b764a299e4504aab87753ddfab3c5, State: Running, Role: LEADER
I20260925 15:22:58.598910 18143 consensus_queue.cc:237] T 7462f50533c24164ace1ee10757282e8 P aa9b764a299e4504aab87753ddfab3c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f06bc4bc858c491b893baed833d7e1cd" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34875 } } peers { permanent_uuid: "aa9b764a299e4504aab87753ddfab3c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 45859 } } peers { permanent_uuid: "0ebeb29ff8794c69802c8ac93d4041f5" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40125 } }
I20260925 15:22:58.600546 17659 catalog_manager.cc:5495] T 7462f50533c24164ace1ee10757282e8 P aa9b764a299e4504aab87753ddfab3c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to aa9b764a299e4504aab87753ddfab3c5 (127.3.211.195). New cstate: current_term: 1 leader_uuid: "aa9b764a299e4504aab87753ddfab3c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f06bc4bc858c491b893baed833d7e1cd" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa9b764a299e4504aab87753ddfab3c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 45859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ebeb29ff8794c69802c8ac93d4041f5" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40125 } health_report { overall_health: UNKNOWN } } }
I20260925 15:22:58.618144 18132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:58.614958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52244 (local address 127.3.211.195:45859)
0925 15:22:58.615054 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:58.615056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:58.615064 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:22:58.615110 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:58.615111 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:58.615130 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:58.615204 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:22:58.615582 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:58.616073 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:58.617297 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:58.617508 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:58.617595 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:58.617798 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:22:58.617912 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:22:58.617975 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:58.618032 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 15:22:58.618052 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 15:22:58.620810 17804 raft_consensus.cc:1261] T 7462f50533c24164ace1ee10757282e8 P f06bc4bc858c491b893baed833d7e1cd [term 1 FOLLOWER]: Refusing update from remote peer aa9b764a299e4504aab87753ddfab3c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:22:58.620810 17938 raft_consensus.cc:1261] T 7462f50533c24164ace1ee10757282e8 P 0ebeb29ff8794c69802c8ac93d4041f5 [term 1 FOLLOWER]: Refusing update from remote peer aa9b764a299e4504aab87753ddfab3c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:22:58.621251 18143 consensus_queue.cc:1035] T 7462f50533c24164ace1ee10757282e8 P aa9b764a299e4504aab87753ddfab3c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f06bc4bc858c491b893baed833d7e1cd" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:22:58.621479 18145 consensus_queue.cc:1035] T 7462f50533c24164ace1ee10757282e8 P aa9b764a299e4504aab87753ddfab3c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebeb29ff8794c69802c8ac93d4041f5" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:22:58.624204 18150 mvcc.cc:205] Tried to move back new op lower bound from 7333272693226405888 to 7333272693141962752. Current Snapshot: MvccSnapshot[applied={T|T < 7333272693226405888}]
I20260925 15:22:58.626226 17658 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:22:58.626341 17658 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:22:58.627421 17658 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68: Sending DeleteTablet for 3 replicas of tablet 7462f50533c24164ace1ee10757282e8
I20260925 15:22:58.627813 18052 tablet_service.cc:1514] Processing DeleteTablet for tablet 7462f50533c24164ace1ee10757282e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:22:58 UTC) from {username='kudu', principal='kudu/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:52242
I20260925 15:22:58.627882 17918 tablet_service.cc:1514] Processing DeleteTablet for tablet 7462f50533c24164ace1ee10757282e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:22:58 UTC) from {username='kudu', principal='kudu/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:50224
I20260925 15:22:58.627909 17784 tablet_service.cc:1514] Processing DeleteTablet for tablet 7462f50533c24164ace1ee10757282e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:22:58 UTC) from {username='kudu', principal='kudu/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:43336
I20260925 15:22:58.628185 18158 tablet_replica.cc:330] T 7462f50533c24164ace1ee10757282e8 P aa9b764a299e4504aab87753ddfab3c5: stopping tablet replica
I20260925 15:22:58.628185 18160 tablet_replica.cc:330] T 7462f50533c24164ace1ee10757282e8 P f06bc4bc858c491b893baed833d7e1cd: stopping tablet replica
I20260925 15:22:58.628201 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 17721
I20260925 15:22:58.628777 18158 raft_consensus.cc:2229] T 7462f50533c24164ace1ee10757282e8 P aa9b764a299e4504aab87753ddfab3c5 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:22:58.628938 18158 raft_consensus.cc:2258] T 7462f50533c24164ace1ee10757282e8 P aa9b764a299e4504aab87753ddfab3c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:22:58.629267 18159 tablet_replica.cc:330] T 7462f50533c24164ace1ee10757282e8 P 0ebeb29ff8794c69802c8ac93d4041f5: stopping tablet replica
I20260925 15:22:58.629410 18159 raft_consensus.cc:2229] T 7462f50533c24164ace1ee10757282e8 P 0ebeb29ff8794c69802c8ac93d4041f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:22:58.629551 18159 raft_consensus.cc:2258] T 7462f50533c24164ace1ee10757282e8 P 0ebeb29ff8794c69802c8ac93d4041f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:22:58.629885 18158 ts_tablet_manager.cc:1905] T 7462f50533c24164ace1ee10757282e8 P aa9b764a299e4504aab87753ddfab3c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:22:58.630033 18159 ts_tablet_manager.cc:1905] T 7462f50533c24164ace1ee10757282e8 P 0ebeb29ff8794c69802c8ac93d4041f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:22:58.631454 18158 ts_tablet_manager.cc:1918] T 7462f50533c24164ace1ee10757282e8 P aa9b764a299e4504aab87753ddfab3c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:22:58.631548 18158 log.cc:1196] T 7462f50533c24164ace1ee10757282e8 P aa9b764a299e4504aab87753ddfab3c5: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-2/wal/wals/7462f50533c24164ace1ee10757282e8
I20260925 15:22:58.631827 18158 ts_tablet_manager.cc:1939] T 7462f50533c24164ace1ee10757282e8 P aa9b764a299e4504aab87753ddfab3c5: Deleting consensus metadata
I20260925 15:22:58.631814 18159 ts_tablet_manager.cc:1918] T 7462f50533c24164ace1ee10757282e8 P 0ebeb29ff8794c69802c8ac93d4041f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:22:58.631903 18159 log.cc:1196] T 7462f50533c24164ace1ee10757282e8 P 0ebeb29ff8794c69802c8ac93d4041f5: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-1/wal/wals/7462f50533c24164ace1ee10757282e8
I20260925 15:22:58.632172 18159 ts_tablet_manager.cc:1939] T 7462f50533c24164ace1ee10757282e8 P 0ebeb29ff8794c69802c8ac93d4041f5: Deleting consensus metadata
I20260925 15:22:58.632306 17646 catalog_manager.cc:4841] TS aa9b764a299e4504aab87753ddfab3c5 (127.3.211.195:45859): tablet 7462f50533c24164ace1ee10757282e8 (table test-table [id=020c9e5f8f744490a58269d119ab4e84]) successfully deleted
I20260925 15:22:58.632598 17645 catalog_manager.cc:4841] TS 0ebeb29ff8794c69802c8ac93d4041f5 (127.3.211.194:40125): tablet 7462f50533c24164ace1ee10757282e8 (table test-table [id=020c9e5f8f744490a58269d119ab4e84]) successfully deleted
W20260925 15:22:58.634030 17645 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.211.193:34875 (error 108)
I20260925 15:22:58.634286 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 17855
I20260925 15:22:58.634568 18129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:58.634447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:34875 (local address 127.0.0.1:43340)
0925 15:22:58.634500 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:58.634529 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.193:34875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 15:22:58.634660 17645 catalog_manager.cc:4584] TS f06bc4bc858c491b893baed833d7e1cd (127.3.211.193:34875): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7462f50533c24164ace1ee10757282e8: Network error: Client connection negotiation failed: client connection to 127.3.211.193:34875: connect: Connection refused (error 111)
I20260925 15:22:58.640728 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 17989
I20260925 15:22:58.646368 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 17627
I20260925 15:22:58.651893 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:34373
--webserver_interface=127.3.211.254
--webserver_port=40273
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:46329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.211.254:34373
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:22:58.739956 18162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:22:58.740490 18162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:22:58.740625 18162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:22:58.742658 18162 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:22:58.742715 18162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:22:58.742733 18162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:22:58.742746 18162 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:22:58.742759 18162 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:22:58.744894 18162 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:46329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.3.211.254:34373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:34373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.211.254
--webserver_port=40273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:22:58.745201 18162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:22:58.745496 18162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:22:58.755566 18171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:58.755566 18169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:58.755550 18168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:22:58.755834 18162 server_base.cc:1034] running on GCE node
I20260925 15:22:58.756102 18162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:22:58.756516 18162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:22:58.757710 18162 hybrid_clock.cc:648] HybridClock initialized: now 1790349778757704 us; error 40 us; skew 500 ppm
Sep 25 15:22:58 dist-test-slave-4bk3 krb5kdc[17614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349778, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:22:58.760620 18162 init.cc:377] Logged in from keytab as kudu/127.3.211.254@KRBTEST.COM (short username kudu)
I20260925 15:22:58.761082 18162 webserver.cc:460] Webserver started at http://127.3.211.254:40273/ using document root <none> and password file <none>
I20260925 15:22:58.761337 18162 fs_manager.cc:359] Metadata directory not provided
I20260925 15:22:58.761395 18162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:22:58.762956 18162 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:22:58.763769 18178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:58.763994 18162 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:22:58.764094 18162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/master-0/wal
uuid: "d18c834f3f6e4638a02dcc3eed10aa68"
format_stamp: "Formatted at 2026-09-25 15:22:57 on dist-test-slave-4bk3"
server_key: "74c164be27e4b8b724c41986d65353ee"
server_key_iv: "726dbd089ad157cbb45e55c653fa7e52"
server_key_version: "encryptionkey@0"
I20260925 15:22:58.764590 18162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:22:58.780659 18162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:22:58.783170 18162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:22:58.783330 18162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:22:58.787457 18162 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.254:34373
I20260925 15:22:58.787513 18230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.254:34373 every 8 connection(s)
I20260925 15:22:58.787945 18162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
I20260925 15:22:58.788537 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 18162
I20260925 15:22:58.788812 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.193:34875
--local_ip_for_outbound_sockets=127.3.211.193
--tserver_master_addrs=127.3.211.254:34373
--webserver_port=36791
--webserver_interface=127.3.211.193
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab
--principal=kudu/127.3.211.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:46329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:22:58.789795 18231 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:22:58.790818 18231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68: Bootstrap starting.
I20260925 15:22:58.799443 18231 log.cc:824] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68: Log is configured to *not* fsync() on all Append() calls
I20260925 15:22:58.801529 18231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:22:58.801841 18231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68: Bootstrap complete.
I20260925 15:22:58.804111 18231 raft_consensus.cc:348] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d18c834f3f6e4638a02dcc3eed10aa68" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 34373 } }
I20260925 15:22:58.804389 18231 raft_consensus.cc:729] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d18c834f3f6e4638a02dcc3eed10aa68, State: Initialized, Role: FOLLOWER
I20260925 15:22:58.804602 18231 consensus_queue.cc:260] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d18c834f3f6e4638a02dcc3eed10aa68" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 34373 } }
I20260925 15:22:58.804698 18231 raft_consensus.cc:388] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:22:58.804757 18231 raft_consensus.cc:482] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:22:58.804804 18231 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68 [term 1 FOLLOWER]: Advancing to term 2
I20260925 15:22:58.805797 18231 raft_consensus.cc:504] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d18c834f3f6e4638a02dcc3eed10aa68" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 34373 } }
I20260925 15:22:58.805917 18231 leader_election.cc:302] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d18c834f3f6e4638a02dcc3eed10aa68; no voters:
I20260925 15:22:58.806118 18231 leader_election.cc:288] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 15:22:58.806156 18235 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 15:22:58.806259 18235 raft_consensus.cc:686] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68 [term 2 LEADER]: Becoming Leader. State: Replica: d18c834f3f6e4638a02dcc3eed10aa68, State: Running, Role: LEADER
I20260925 15:22:58.806389 18231 sys_catalog.cc:564] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:22:58.806417 18235 consensus_queue.cc:237] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d18c834f3f6e4638a02dcc3eed10aa68" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 34373 } }
I20260925 15:22:58.806756 18236 sys_catalog.cc:455] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d18c834f3f6e4638a02dcc3eed10aa68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d18c834f3f6e4638a02dcc3eed10aa68" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 34373 } } }
I20260925 15:22:58.806769 18237 sys_catalog.cc:455] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d18c834f3f6e4638a02dcc3eed10aa68. Latest consensus state: current_term: 2 leader_uuid: "d18c834f3f6e4638a02dcc3eed10aa68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d18c834f3f6e4638a02dcc3eed10aa68" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 34373 } } }
I20260925 15:22:58.806851 18236 sys_catalog.cc:458] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68 [sys.catalog]: This master's current role is: LEADER
I20260925 15:22:58.806869 18237 sys_catalog.cc:458] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68 [sys.catalog]: This master's current role is: LEADER
I20260925 15:22:58.807047 18241 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:22:58.808086 18241 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:22:58.808282 18241 catalog_manager.cc:1259] Loaded cluster ID: 331dfeb1e768466396a7dee70f75b8e6
I20260925 15:22:58.808593 18241 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:22:58.808838 18241 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 15:22:58.811585 18241 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 15:22:58.814453 18241 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:22:58.814677 18241 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68: Loaded TSK: 0
I20260925 15:22:58.814888 18241 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 15:22:58.889482 18233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:22:58.889753 18233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:22:58.889793 18233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:22:58.891053 18233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:22:58.891155 18233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:22:58.891211 18233 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.193
I20260925 15:22:58.892802 18233 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:46329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab
--principal=kudu/127.3.211.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.193:34875
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.211.193
--webserver_port=36791
--tserver_master_addrs=127.3.211.254:34373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.193
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:22:58.893103 18233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:22:58.893354 18233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:22:58.898769 18259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:58.898792 18258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:58.898769 18261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:22:58.899550 18233 server_base.cc:1034] running on GCE node
I20260925 15:22:58.899796 18233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:22:58.900108 18233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:22:58.901304 18233 hybrid_clock.cc:648] HybridClock initialized: now 1790349778901284 us; error 36 us; skew 500 ppm
Sep 25 15:22:58 dist-test-slave-4bk3 krb5kdc[17614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349778, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:22:58.904788 18233 init.cc:377] Logged in from keytab as kudu/127.3.211.193@KRBTEST.COM (short username kudu)
I20260925 15:22:58.905521 18233 webserver.cc:460] Webserver started at http://127.3.211.193:36791/ using document root <none> and password file <none>
I20260925 15:22:58.905817 18233 fs_manager.cc:359] Metadata directory not provided
I20260925 15:22:58.905913 18233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:22:58.908414 18233 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:22:58.909425 18268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:58.909696 18233 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:22:58.909801 18233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-0/wal
uuid: "f06bc4bc858c491b893baed833d7e1cd"
format_stamp: "Formatted at 2026-09-25 15:22:58 on dist-test-slave-4bk3"
server_key: "fd1d7454747e3eac10ae4b0f703fa59a"
server_key_iv: "d9b085bd2ead1a20cb2e76205e491cd0"
server_key_version: "encryptionkey@0"
I20260925 15:22:58.910302 18233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:22:58.941824 18233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:22:58.944957 18233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:22:58.945173 18233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:22:58.945489 18233 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:22:58.946147 18275 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 15:22:58.947904 18233 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 15:22:58.947983 18233 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 15:22:58.948136 18233 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 15:22:58.949105 18233 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 15:22:58.949155 18233 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 15:22:58.949288 18275 tablet_bootstrap.cc:492] T 7462f50533c24164ace1ee10757282e8 P f06bc4bc858c491b893baed833d7e1cd: Bootstrap starting.
I20260925 15:22:58.957237 18233 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.193:34875
I20260925 15:22:58.957319 18382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.193:34875 every 8 connection(s)
I20260925 15:22:58.957805 18233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:22:58.961032 18275 log.cc:824] T 7462f50533c24164ace1ee10757282e8 P f06bc4bc858c491b893baed833d7e1cd: Log is configured to *not* fsync() on all Append() calls
I20260925 15:22:58.963820 18275 tablet_bootstrap.cc:492] T 7462f50533c24164ace1ee10757282e8 P f06bc4bc858c491b893baed833d7e1cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:22:58.964174 18275 tablet_bootstrap.cc:492] T 7462f50533c24164ace1ee10757282e8 P f06bc4bc858c491b893baed833d7e1cd: Bootstrap complete.
I20260925 15:22:58.965385 18275 ts_tablet_manager.cc:1397] T 7462f50533c24164ace1ee10757282e8 P f06bc4bc858c491b893baed833d7e1cd: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.007s
I20260925 15:22:58.965452 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 18233
I20260925 15:22:58.965730 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.194:40125
--local_ip_for_outbound_sockets=127.3.211.194
--tserver_master_addrs=127.3.211.254:34373
--webserver_port=36343
--webserver_interface=127.3.211.194
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab
--principal=kudu/127.3.211.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:46329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:22:58.967478 18275 raft_consensus.cc:348] T 7462f50533c24164ace1ee10757282e8 P f06bc4bc858c491b893baed833d7e1cd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f06bc4bc858c491b893baed833d7e1cd" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34875 } } peers { permanent_uuid: "aa9b764a299e4504aab87753ddfab3c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 45859 } } peers { permanent_uuid: "0ebeb29ff8794c69802c8ac93d4041f5" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40125 } }
I20260925 15:22:58.967761 18275 raft_consensus.cc:729] T 7462f50533c24164ace1ee10757282e8 P f06bc4bc858c491b893baed833d7e1cd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f06bc4bc858c491b893baed833d7e1cd, State: Initialized, Role: FOLLOWER
Sep 25 15:22:58 dist-test-slave-4bk3 krb5kdc[17614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349778, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.193@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:22:58.967900 18275 consensus_queue.cc:260] T 7462f50533c24164ace1ee10757282e8 P f06bc4bc858c491b893baed833d7e1cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f06bc4bc858c491b893baed833d7e1cd" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34875 } } peers { permanent_uuid: "aa9b764a299e4504aab87753ddfab3c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 45859 } } peers { permanent_uuid: "0ebeb29ff8794c69802c8ac93d4041f5" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40125 } }
I20260925 15:22:58.968235 18275 ts_tablet_manager.cc:1428] T 7462f50533c24164ace1ee10757282e8 P f06bc4bc858c491b893baed833d7e1cd: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260925 15:22:58.973161 18386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:58.958680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:59391 (local address 127.3.211.254:34373)
0925 15:22:58.960538 (+ 1858us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:58.960550 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:58.960605 (+ 55us) server_negotiation.cc:407] Connection header received
0925 15:22:58.960745 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:58.960751 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:58.960804 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:58.960990 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:58.961811 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:58.963190 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:58.964699 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:58.965425 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:58.965547 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:58.968573 (+ 3026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:58.968578 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:58.968578 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:58.968589 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:58.970750 (+ 2161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:58.971347 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:58.971351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:58.971351 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:58.971489 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:58.971832 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:58.971833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:58.971833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:58.972311 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:58.972428 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:58.972804 (+ 376us) server_negotiation.cc:299] Negotiation successful
0925 15:22:58.972948 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1795,"thread_start_us":88,"threads_started":1}
I20260925 15:22:58.973237 18385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:58.958566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.193:59391)
0925 15:22:58.958753 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:58.958798 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:58.959614 (+ 816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:58.961030 (+ 1416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:58.961035 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:58.961404 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:22:58.961669 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:58.961678 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:58.963338 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:58.963339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:58.964599 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:58.964602 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:58.965691 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:58.965692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:58.965829 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:58.966472 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:22:58.966491 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:22:58.968401 (+ 1910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:22:58.970868 (+ 2467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:58.970876 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:58.970879 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:58.971233 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:58.971576 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:58.971577 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:58.971578 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:58.971737 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:58.972407 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:22:58.972414 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:22:58.972593 (+ 179us) client_negotiation.cc:769] Sending connection context
0925 15:22:58.972731 (+ 138us) client_negotiation.cc:240] Negotiation successful
0925 15:22:58.972950 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":110,"threads_started":1}
I20260925 15:22:58.974435 18383 heartbeater.cc:347] Connected to a master server at 127.3.211.254:34373
I20260925 15:22:58.974558 18383 heartbeater.cc:464] Registering TS with master...
I20260925 15:22:58.974793 18383 heartbeater.cc:511] Master 127.3.211.254:34373 requested a full tablet report, sending...
I20260925 15:22:58.975721 18195 ts_manager.cc:194] Registered new tserver with Master: f06bc4bc858c491b893baed833d7e1cd (127.3.211.193:34875)
I20260925 15:22:58.976914 18195 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.193@KRBTEST.COM'} at 127.3.211.193:59391
I20260925 15:22:58.978034 18383 heartbeater.cc:503] Master 127.3.211.254:34373 was elected leader, sending a full tablet report...
Sep 25 15:22:58 dist-test-slave-4bk3 krb5kdc[17614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349778, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.254@KRBTEST.COM for kudu/127.3.211.193@KRBTEST.COM
I20260925 15:22:58.988332 18391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:58.976665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:34875 (local address 127.0.0.1:43350)
0925 15:22:58.976949 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:58.977001 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:58.977105 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:58.978320 (+ 1215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:58.978327 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:58.978729 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:22:58.978803 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:58.978806 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:58.980118 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:58.980119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:58.981241 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:58.981244 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:58.981983 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:58.981985 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:58.982061 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:58.982786 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:22:58.982792 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:22:58.984578 (+ 1786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:22:58.987055 (+ 2477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:58.987064 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:58.987067 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:58.987319 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:58.987577 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:58.987579 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:58.987579 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:58.987639 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:58.987973 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:22:58.987981 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:22:58.988033 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 15:22:58.988115 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 15:22:58.988251 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":63,"threads_started":1}
I20260925 15:22:58.988425 18390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:58.976495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43350 (local address 127.3.211.193:34875)
0925 15:22:58.976878 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:58.976884 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:58.977260 (+ 376us) server_negotiation.cc:407] Connection header received
0925 15:22:58.977302 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:58.977305 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:58.977321 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:58.978239 (+ 918us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:58.978896 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:58.980016 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:58.981337 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:58.981888 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:58.981972 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:58.984708 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:58.984720 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:58.984720 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:58.984729 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:58.986775 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:58.987418 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:58.987422 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:58.987423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:58.987493 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:58.987702 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:58.987703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:58.987703 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:58.987891 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:58.987989 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:58.988177 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 15:22:58.988273 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":89,"threads_started":1}
I20260925 15:22:58.988919 18317 tablet_service.cc:1514] Processing DeleteTablet for tablet 7462f50533c24164ace1ee10757282e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:22:58 UTC) from {username='kudu', principal='kudu/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:43350
I20260925 15:22:58.989673 18392 tablet_replica.cc:330] stopping tablet replica
I20260925 15:22:58.989794 18392 raft_consensus.cc:2229] T 7462f50533c24164ace1ee10757282e8 P f06bc4bc858c491b893baed833d7e1cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:22:58.989894 18392 raft_consensus.cc:2258] T 7462f50533c24164ace1ee10757282e8 P f06bc4bc858c491b893baed833d7e1cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:22:58.990221 18392 ts_tablet_manager.cc:1905] T 7462f50533c24164ace1ee10757282e8 P f06bc4bc858c491b893baed833d7e1cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:22:58.992475 18392 ts_tablet_manager.cc:1918] T 7462f50533c24164ace1ee10757282e8 P f06bc4bc858c491b893baed833d7e1cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:22:58.992568 18392 log.cc:1196] T 7462f50533c24164ace1ee10757282e8 P f06bc4bc858c491b893baed833d7e1cd: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-0/wal/wals/7462f50533c24164ace1ee10757282e8
I20260925 15:22:58.992835 18392 ts_tablet_manager.cc:1939] T 7462f50533c24164ace1ee10757282e8 P f06bc4bc858c491b893baed833d7e1cd: Deleting consensus metadata
I20260925 15:22:58.993429 18181 catalog_manager.cc:4841] TS f06bc4bc858c491b893baed833d7e1cd (127.3.211.193:34875): tablet 7462f50533c24164ace1ee10757282e8 (table test-table [id=020c9e5f8f744490a58269d119ab4e84]) successfully deleted
W20260925 15:22:59.060798 18388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:22:59.061033 18388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:22:59.061074 18388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:22:59.062291 18388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:22:59.062368 18388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:22:59.062414 18388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.194
I20260925 15:22:59.063946 18388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:46329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab
--principal=kudu/127.3.211.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.194:40125
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.211.194
--webserver_port=36343
--tserver_master_addrs=127.3.211.254:34373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.194
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:22:59.064224 18388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:22:59.064522 18388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:22:59.069615 18397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:59.069677 18398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:22:59.069717 18388 server_base.cc:1034] running on GCE node
W20260925 15:22:59.069674 18400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:22:59.070108 18388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:22:59.070386 18388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:22:59.071553 18388 hybrid_clock.cc:648] HybridClock initialized: now 1790349779071542 us; error 27 us; skew 500 ppm
Sep 25 15:22:59 dist-test-slave-4bk3 krb5kdc[17614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349779, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:22:59.074316 18388 init.cc:377] Logged in from keytab as kudu/127.3.211.194@KRBTEST.COM (short username kudu)
I20260925 15:22:59.074860 18388 webserver.cc:460] Webserver started at http://127.3.211.194:36343/ using document root <none> and password file <none>
I20260925 15:22:59.075101 18388 fs_manager.cc:359] Metadata directory not provided
I20260925 15:22:59.075186 18388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:22:59.076659 18388 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:22:59.077383 18407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:59.077579 18388 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 15:22:59.077687 18388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-1/wal
uuid: "0ebeb29ff8794c69802c8ac93d4041f5"
format_stamp: "Formatted at 2026-09-25 15:22:58 on dist-test-slave-4bk3"
server_key: "f3d329eae3527319e15752293c4cf17d"
server_key_iv: "24d7c8213763f7105ea359c64b904acf"
server_key_version: "encryptionkey@0"
I20260925 15:22:59.078040 18388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:22:59.087121 18388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:22:59.089354 18388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:22:59.089535 18388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:22:59.089785 18388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:22:59.090108 18388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:22:59.090142 18388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:59.090209 18388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:22:59.090238 18388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:59.096626 18388 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.194:40125
I20260925 15:22:59.096725 18520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.194:40125 every 8 connection(s)
I20260925 15:22:59.097127 18388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
Sep 25 15:22:59 dist-test-slave-4bk3 krb5kdc[17614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349779, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.194@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:22:59.103662 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 18388
I20260925 15:22:59.103958 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.195:45859
--local_ip_for_outbound_sockets=127.3.211.195
--tserver_master_addrs=127.3.211.254:34373
--webserver_port=41357
--webserver_interface=127.3.211.195
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab
--principal=kudu/127.3.211.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:46329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:22:59.107671 18386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:59.097984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:34445 (local address 127.3.211.254:34373)
0925 15:22:59.098028 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:59.098030 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:59.098248 (+ 218us) server_negotiation.cc:407] Connection header received
0925 15:22:59.098762 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:59.098763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:59.098778 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:59.098850 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:59.099791 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:59.100285 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:59.101377 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:59.101597 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:59.101654 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:59.104220 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:59.104221 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:59.104222 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:59.104230 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:59.105833 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:59.106318 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:59.106319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:59.106320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:59.106388 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:59.106789 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:59.106790 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:59.106790 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:59.106913 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:59.106983 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:59.107385 (+ 402us) server_negotiation.cc:299] Negotiation successful
0925 15:22:59.107533 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 15:22:59.107949 18523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:59.097976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.194:34445)
0925 15:22:59.098118 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:59.098154 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:59.098647 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:59.098905 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:59.098908 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:59.099498 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:22:59.099708 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:59.099713 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:59.100470 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:59.100471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:59.101291 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:59.101294 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:59.101698 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:59.101699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:59.101823 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:59.102432 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:22:59.102441 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:22:59.104084 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:22:59.105956 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:59.105967 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:59.105970 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:59.106230 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:59.106498 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:59.106499 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:59.106499 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:59.106712 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:59.106990 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:22:59.106998 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:22:59.107193 (+ 195us) client_negotiation.cc:769] Sending connection context
0925 15:22:59.107330 (+ 137us) client_negotiation.cc:240] Negotiation successful
0925 15:22:59.107620 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":76,"threads_started":1}
I20260925 15:22:59.108497 18521 heartbeater.cc:347] Connected to a master server at 127.3.211.254:34373
I20260925 15:22:59.108639 18521 heartbeater.cc:464] Registering TS with master...
I20260925 15:22:59.108850 18521 heartbeater.cc:511] Master 127.3.211.254:34373 requested a full tablet report, sending...
I20260925 15:22:59.109406 18195 ts_manager.cc:194] Registered new tserver with Master: 0ebeb29ff8794c69802c8ac93d4041f5 (127.3.211.194:40125)
I20260925 15:22:59.109941 18195 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.194@KRBTEST.COM'} at 127.3.211.194:34445
W20260925 15:22:59.191154 18524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:22:59.191401 18524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:22:59.191427 18524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:22:59.192994 18524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:22:59.193066 18524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:22:59.193102 18524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.195
I20260925 15:22:59.194744 18524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:46329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab
--principal=kudu/127.3.211.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.195:45859
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.211.195
--webserver_port=41357
--tserver_master_addrs=127.3.211.254:34373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.195
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:22:59.195043 18524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:22:59.195302 18524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:22:59.200733 18530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:59.200744 18532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:59.200760 18529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:22:59.201155 18524 server_base.cc:1034] running on GCE node
I20260925 15:22:59.201392 18524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:22:59.201620 18524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:22:59.202766 18524 hybrid_clock.cc:648] HybridClock initialized: now 1790349779202758 us; error 34 us; skew 500 ppm
Sep 25 15:22:59 dist-test-slave-4bk3 krb5kdc[17614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349779, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:22:59.205582 18524 init.cc:377] Logged in from keytab as kudu/127.3.211.195@KRBTEST.COM (short username kudu)
I20260925 15:22:59.206171 18524 webserver.cc:460] Webserver started at http://127.3.211.195:41357/ using document root <none> and password file <none>
I20260925 15:22:59.206418 18524 fs_manager.cc:359] Metadata directory not provided
I20260925 15:22:59.206502 18524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:22:59.208160 18524 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:22:59.208871 18539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:59.209044 18524 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 15:22:59.209159 18524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-2/wal
uuid: "aa9b764a299e4504aab87753ddfab3c5"
format_stamp: "Formatted at 2026-09-25 15:22:58 on dist-test-slave-4bk3"
server_key: "73509fac2ace0aa24b5499535efcf550"
server_key_iv: "886bf768a689649c7627c90016fd9a0f"
server_key_version: "encryptionkey@0"
I20260925 15:22:59.209553 18524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:22:59.238170 18524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:22:59.241004 18524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:22:59.241169 18524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:22:59.241571 18524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:22:59.241925 18524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:22:59.241966 18524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:59.242023 18524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:22:59.242051 18524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:59.248106 18524 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.195:45859
I20260925 15:22:59.248147 18652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.195:45859 every 8 connection(s)
I20260925 15:22:59.248543 18524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
I20260925 15:22:59.250258 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 18524
Sep 25 15:22:59 dist-test-slave-4bk3 krb5kdc[17614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349779, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.195@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:22:59.259563 18657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:59.250723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39468 (local address 127.3.211.254:34373)
0925 15:22:59.250821 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:59.250828 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:59.251157 (+ 329us) server_negotiation.cc:407] Connection header received
0925 15:22:59.251282 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:59.251283 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:59.251295 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:59.251374 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:59.252481 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:59.253384 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:59.254539 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:59.254746 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:59.254856 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:59.256428 (+ 1572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:59.256429 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:59.256429 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:59.256435 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:59.258265 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:59.258691 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:59.258693 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:59.258693 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:59.258762 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:59.258989 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:59.258989 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:59.258990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:59.259116 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:59.259305 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:59.259342 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 15:22:59.259436 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":76,"thread_start_us":57,"threads_started":1}
I20260925 15:22:59.260957 18655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:59.249553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.195:49823)
0925 15:22:59.249733 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:59.249776 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:59.250255 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:59.250628 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:59.250633 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:59.252145 (+ 1512us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:22:59.252520 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:59.252527 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:59.253547 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:59.253548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:59.254733 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:59.254736 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:59.255137 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:59.255138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:59.255258 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:59.255743 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:22:59.255755 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:22:59.257580 (+ 1825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:22:59.259325 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:59.259335 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:59.259338 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:59.259666 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:59.259917 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:22:59.259918 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:22:59.259918 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:22:59.260071 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:22:59.260302 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:22:59.260309 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:22:59.260521 (+ 212us) client_negotiation.cc:769] Sending connection context
0925 15:22:59.260631 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 15:22:59.260771 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":111,"threads_started":1}
I20260925 15:22:59.261092 18386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:59.249560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:49823 (local address 127.3.211.254:34373)
0925 15:22:59.249642 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:59.249644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:59.249868 (+ 224us) server_negotiation.cc:407] Connection header received
0925 15:22:59.250377 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:59.250379 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:59.250394 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:59.250476 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:22:59.252703 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:59.253384 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:59.254824 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:59.255043 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:59.255096 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:59.257681 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:22:59.257683 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:22:59.257684 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:22:59.257691 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:22:59.259201 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:59.259750 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:59.259751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:59.259751 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:59.259806 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:22:59.260135 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:22:59.260136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:22:59.260136 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:22:59.260237 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:22:59.260288 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:59.260784 (+ 496us) server_negotiation.cc:299] Negotiation successful
0925 15:22:59.260925 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 15:22:59.261577 18653 heartbeater.cc:347] Connected to a master server at 127.3.211.254:34373
I20260925 15:22:59.261685 18653 heartbeater.cc:464] Registering TS with master...
I20260925 15:22:59.261894 18653 heartbeater.cc:511] Master 127.3.211.254:34373 requested a full tablet report, sending...
I20260925 15:22:59.262429 18195 ts_manager.cc:194] Registered new tserver with Master: aa9b764a299e4504aab87753ddfab3c5 (127.3.211.195:45859)
I20260925 15:22:59.262850 18195 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.195@KRBTEST.COM'} at 127.3.211.195:49823
I20260925 15:22:59.272010 3919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:22:59.275336 18386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:59.272492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39474 (local address 127.3.211.254:34373)
0925 15:22:59.272565 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:59.272567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:59.272667 (+ 100us) server_negotiation.cc:407] Connection header received
0925 15:22:59.272741 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:59.272742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:59.272780 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:59.272844 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:22:59.272943 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:59.273423 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:59.274243 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:59.274447 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:59.274530 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:59.274656 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:22:59.274929 (+ 273us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:22:59.275080 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:59.275180 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 15:22:59.275227 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 15:22:59.275823 18195 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:39474:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:22:59.278422 18195 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:22:59.282074 18317 tablet_service.cc:1467] Processing CreateTablet for tablet c30983c7ec7c4af6be799d0cf985cc6c (DEFAULT_TABLE table=test-table [id=952db465628b4a67b12983aebd74fdf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:22:59.282312 18317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c30983c7ec7c4af6be799d0cf985cc6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:22:59.284101 18275 tablet_bootstrap.cc:492] T c30983c7ec7c4af6be799d0cf985cc6c P f06bc4bc858c491b893baed833d7e1cd: Bootstrap starting.
I20260925 15:22:59.284561 18275 tablet_bootstrap.cc:654] T c30983c7ec7c4af6be799d0cf985cc6c P f06bc4bc858c491b893baed833d7e1cd: Neither blocks nor log segments found. Creating new log.
I20260925 15:22:59.285416 18660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:59.281310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52252 (local address 127.3.211.195:45859)
0925 15:22:59.281523 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:59.281527 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:59.281538 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:22:59.281581 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:59.281583 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:59.281607 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:59.281720 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:22:59.281971 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:59.282947 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:59.284280 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:59.284983 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:59.285112 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:59.285195 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:59.285311 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 15:22:59.285348 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":80,"threads_started":1}
I20260925 15:22:59.285434 18391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:59.281058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.195:45859 (local address 127.0.0.1:52252)
0925 15:22:59.281147 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:59.281159 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:59.281210 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:59.281740 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:59.281741 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:59.281743 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:22:59.281804 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:59.281806 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:59.283086 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:59.283087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:59.283873 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:59.283876 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:59.285118 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:59.285119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:59.285194 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:59.285223 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 15:22:59.285297 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 15:22:59.285353 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 15:22:59.285835 18661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:59.281061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.194:40125 (local address 127.0.0.1:50230)
0925 15:22:59.281662 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:59.281672 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:59.281726 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:59.281970 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:59.281971 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:59.281974 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:22:59.282022 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:59.282024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:59.283397 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:59.283399 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:59.284530 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:59.284534 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:59.285583 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:59.285584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:59.285653 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:59.285657 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:22:59.285716 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 15:22:59.285757 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"mutex_wait_us":292,"thread_start_us":53,"threads_started":1}
I20260925 15:22:59.285902 18659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:59.281153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50230 (local address 127.3.211.194:40125)
0925 15:22:59.281482 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:59.281488 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:59.281783 (+ 295us) server_negotiation.cc:407] Connection header received
0925 15:22:59.281827 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:59.281829 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:59.281846 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:59.281928 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:22:59.282248 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:59.283269 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:59.284653 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:59.285410 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:59.285476 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:59.285531 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:59.285762 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 15:22:59.285816 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":130,"threads_started":1}
I20260925 15:22:59.285997 18275 tablet_bootstrap.cc:492] T c30983c7ec7c4af6be799d0cf985cc6c P f06bc4bc858c491b893baed833d7e1cd: No bootstrap required, opened a new log
I20260925 15:22:59.286057 18275 ts_tablet_manager.cc:1397] T c30983c7ec7c4af6be799d0cf985cc6c P f06bc4bc858c491b893baed833d7e1cd: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 15:22:59.286302 18587 tablet_service.cc:1467] Processing CreateTablet for tablet c30983c7ec7c4af6be799d0cf985cc6c (DEFAULT_TABLE table=test-table [id=952db465628b4a67b12983aebd74fdf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:22:59.286475 18275 raft_consensus.cc:348] T c30983c7ec7c4af6be799d0cf985cc6c P f06bc4bc858c491b893baed833d7e1cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebeb29ff8794c69802c8ac93d4041f5" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40125 } } peers { permanent_uuid: "aa9b764a299e4504aab87753ddfab3c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 45859 } } peers { permanent_uuid: "f06bc4bc858c491b893baed833d7e1cd" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34875 } }
I20260925 15:22:59.286566 18275 raft_consensus.cc:374] T c30983c7ec7c4af6be799d0cf985cc6c P f06bc4bc858c491b893baed833d7e1cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:22:59.286623 18275 raft_consensus.cc:729] T c30983c7ec7c4af6be799d0cf985cc6c P f06bc4bc858c491b893baed833d7e1cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f06bc4bc858c491b893baed833d7e1cd, State: Initialized, Role: FOLLOWER
I20260925 15:22:59.286674 18275 consensus_queue.cc:260] T c30983c7ec7c4af6be799d0cf985cc6c P f06bc4bc858c491b893baed833d7e1cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebeb29ff8794c69802c8ac93d4041f5" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40125 } } peers { permanent_uuid: "aa9b764a299e4504aab87753ddfab3c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 45859 } } peers { permanent_uuid: "f06bc4bc858c491b893baed833d7e1cd" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34875 } }
I20260925 15:22:59.286708 18587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c30983c7ec7c4af6be799d0cf985cc6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:22:59.286779 18275 ts_tablet_manager.cc:1428] T c30983c7ec7c4af6be799d0cf985cc6c P f06bc4bc858c491b893baed833d7e1cd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 15:22:59.286899 18455 tablet_service.cc:1467] Processing CreateTablet for tablet c30983c7ec7c4af6be799d0cf985cc6c (DEFAULT_TABLE table=test-table [id=952db465628b4a67b12983aebd74fdf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:22:59.287243 18455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c30983c7ec7c4af6be799d0cf985cc6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:22:59.290760 18663 tablet_bootstrap.cc:492] T c30983c7ec7c4af6be799d0cf985cc6c P aa9b764a299e4504aab87753ddfab3c5: Bootstrap starting.
I20260925 15:22:59.290987 18664 tablet_bootstrap.cc:492] T c30983c7ec7c4af6be799d0cf985cc6c P 0ebeb29ff8794c69802c8ac93d4041f5: Bootstrap starting.
I20260925 15:22:59.291594 18664 tablet_bootstrap.cc:654] T c30983c7ec7c4af6be799d0cf985cc6c P 0ebeb29ff8794c69802c8ac93d4041f5: Neither blocks nor log segments found. Creating new log.
I20260925 15:22:59.292253 18664 log.cc:824] T c30983c7ec7c4af6be799d0cf985cc6c P 0ebeb29ff8794c69802c8ac93d4041f5: Log is configured to *not* fsync() on all Append() calls
I20260925 15:22:59.293128 18663 tablet_bootstrap.cc:654] T c30983c7ec7c4af6be799d0cf985cc6c P aa9b764a299e4504aab87753ddfab3c5: Neither blocks nor log segments found. Creating new log.
I20260925 15:22:59.293160 18664 tablet_bootstrap.cc:492] T c30983c7ec7c4af6be799d0cf985cc6c P 0ebeb29ff8794c69802c8ac93d4041f5: No bootstrap required, opened a new log
I20260925 15:22:59.293229 18664 ts_tablet_manager.cc:1397] T c30983c7ec7c4af6be799d0cf985cc6c P 0ebeb29ff8794c69802c8ac93d4041f5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 15:22:59.293845 18663 log.cc:824] T c30983c7ec7c4af6be799d0cf985cc6c P aa9b764a299e4504aab87753ddfab3c5: Log is configured to *not* fsync() on all Append() calls
I20260925 15:22:59.294684 18663 tablet_bootstrap.cc:492] T c30983c7ec7c4af6be799d0cf985cc6c P aa9b764a299e4504aab87753ddfab3c5: No bootstrap required, opened a new log
I20260925 15:22:59.294780 18663 ts_tablet_manager.cc:1397] T c30983c7ec7c4af6be799d0cf985cc6c P aa9b764a299e4504aab87753ddfab3c5: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 15:22:59.295229 18664 raft_consensus.cc:348] T c30983c7ec7c4af6be799d0cf985cc6c P 0ebeb29ff8794c69802c8ac93d4041f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebeb29ff8794c69802c8ac93d4041f5" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40125 } } peers { permanent_uuid: "aa9b764a299e4504aab87753ddfab3c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 45859 } } peers { permanent_uuid: "f06bc4bc858c491b893baed833d7e1cd" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34875 } }
I20260925 15:22:59.295410 18664 raft_consensus.cc:374] T c30983c7ec7c4af6be799d0cf985cc6c P 0ebeb29ff8794c69802c8ac93d4041f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:22:59.295454 18664 raft_consensus.cc:729] T c30983c7ec7c4af6be799d0cf985cc6c P 0ebeb29ff8794c69802c8ac93d4041f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ebeb29ff8794c69802c8ac93d4041f5, State: Initialized, Role: FOLLOWER
I20260925 15:22:59.295648 18664 consensus_queue.cc:260] T c30983c7ec7c4af6be799d0cf985cc6c P 0ebeb29ff8794c69802c8ac93d4041f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebeb29ff8794c69802c8ac93d4041f5" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40125 } } peers { permanent_uuid: "aa9b764a299e4504aab87753ddfab3c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 45859 } } peers { permanent_uuid: "f06bc4bc858c491b893baed833d7e1cd" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34875 } }
I20260925 15:22:59.295962 18664 ts_tablet_manager.cc:1428] T c30983c7ec7c4af6be799d0cf985cc6c P 0ebeb29ff8794c69802c8ac93d4041f5: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 15:22:59.296025 18521 heartbeater.cc:503] Master 127.3.211.254:34373 was elected leader, sending a full tablet report...
I20260925 15:22:59.296737 18663 raft_consensus.cc:348] T c30983c7ec7c4af6be799d0cf985cc6c P aa9b764a299e4504aab87753ddfab3c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebeb29ff8794c69802c8ac93d4041f5" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40125 } } peers { permanent_uuid: "aa9b764a299e4504aab87753ddfab3c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 45859 } } peers { permanent_uuid: "f06bc4bc858c491b893baed833d7e1cd" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34875 } }
I20260925 15:22:59.296902 18663 raft_consensus.cc:374] T c30983c7ec7c4af6be799d0cf985cc6c P aa9b764a299e4504aab87753ddfab3c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:22:59.296952 18663 raft_consensus.cc:729] T c30983c7ec7c4af6be799d0cf985cc6c P aa9b764a299e4504aab87753ddfab3c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa9b764a299e4504aab87753ddfab3c5, State: Initialized, Role: FOLLOWER
I20260925 15:22:59.297070 18663 consensus_queue.cc:260] T c30983c7ec7c4af6be799d0cf985cc6c P aa9b764a299e4504aab87753ddfab3c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebeb29ff8794c69802c8ac93d4041f5" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40125 } } peers { permanent_uuid: "aa9b764a299e4504aab87753ddfab3c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 45859 } } peers { permanent_uuid: "f06bc4bc858c491b893baed833d7e1cd" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34875 } }
I20260925 15:22:59.297369 18663 ts_tablet_manager.cc:1428] T c30983c7ec7c4af6be799d0cf985cc6c P aa9b764a299e4504aab87753ddfab3c5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:22:59.297366 18653 heartbeater.cc:503] Master 127.3.211.254:34373 was elected leader, sending a full tablet report...
W20260925 15:22:59.347781 18522 tablet.cc:2369] T c30983c7ec7c4af6be799d0cf985cc6c P 0ebeb29ff8794c69802c8ac93d4041f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:22:59.410283 18667 raft_consensus.cc:482] T c30983c7ec7c4af6be799d0cf985cc6c P 0ebeb29ff8794c69802c8ac93d4041f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:22:59.410416 18667 raft_consensus.cc:504] T c30983c7ec7c4af6be799d0cf985cc6c P 0ebeb29ff8794c69802c8ac93d4041f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebeb29ff8794c69802c8ac93d4041f5" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40125 } } peers { permanent_uuid: "aa9b764a299e4504aab87753ddfab3c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 45859 } } peers { permanent_uuid: "f06bc4bc858c491b893baed833d7e1cd" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34875 } }
I20260925 15:22:59.410776 18667 leader_election.cc:288] T c30983c7ec7c4af6be799d0cf985cc6c P 0ebeb29ff8794c69802c8ac93d4041f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aa9b764a299e4504aab87753ddfab3c5 (127.3.211.195:45859), f06bc4bc858c491b893baed833d7e1cd (127.3.211.193:34875)
I20260925 15:22:59.414129 18660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:59.411097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:41443 (local address 127.3.211.195:45859)
0925 15:22:59.411233 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:59.411235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:59.411282 (+ 47us) server_negotiation.cc:407] Connection header received
0925 15:22:59.411373 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:59.411374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:59.411384 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:59.411446 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:22:59.411627 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:59.412082 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:59.413334 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:59.413730 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:59.413845 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:59.413855 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:59.414015 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 15:22:59.414055 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260925 15:22:59.414160 18669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:59.411017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.195:45859 (local address 127.3.211.194:41443)
0925 15:22:59.411206 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:59.411218 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:59.411299 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:59.411455 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:59.411456 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:59.411457 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:22:59.411552 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:59.411554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:59.412219 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:59.412220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:59.413172 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:59.413175 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:59.413847 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:59.413849 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:59.413928 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:59.413933 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:22:59.413990 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 15:22:59.414034 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":33,"thread_start_us":114,"threads_started":1}
I20260925 15:22:59.414290 18523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:59.411018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:34875 (local address 127.3.211.194:46533)
0925 15:22:59.411080 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:59.411093 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:59.411156 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:59.411368 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:22:59.411369 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:22:59.411370 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:22:59.411447 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:59.411449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:59.412159 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:59.412161 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:59.413168 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:22:59.413171 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:22:59.414021 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:22:59.414022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:22:59.414101 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:59.414105 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:22:59.414169 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 15:22:59.414216 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":75}
I20260925 15:22:59.414324 18390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:59.411098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:46533 (local address 127.3.211.193:34875)
0925 15:22:59.411171 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:59.411174 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:59.411185 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:22:59.411224 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:59.411225 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:59.411256 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:59.411331 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:22:59.411538 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:59.412068 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:59.413325 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:59.413905 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:59.413986 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:59.414058 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:59.414179 (+ 121us) server_negotiation.cc:299] Negotiation successful
0925 15:22:59.414248 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 15:22:59.414614 18332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c30983c7ec7c4af6be799d0cf985cc6c" candidate_uuid: "0ebeb29ff8794c69802c8ac93d4041f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f06bc4bc858c491b893baed833d7e1cd" is_pre_election: true
I20260925 15:22:59.414775 18607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c30983c7ec7c4af6be799d0cf985cc6c" candidate_uuid: "0ebeb29ff8794c69802c8ac93d4041f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa9b764a299e4504aab87753ddfab3c5" is_pre_election: true
I20260925 15:22:59.414819 18332 raft_consensus.cc:2454] T c30983c7ec7c4af6be799d0cf985cc6c P f06bc4bc858c491b893baed833d7e1cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebeb29ff8794c69802c8ac93d4041f5 in term 0.
I20260925 15:22:59.414906 18607 raft_consensus.cc:2454] T c30983c7ec7c4af6be799d0cf985cc6c P aa9b764a299e4504aab87753ddfab3c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebeb29ff8794c69802c8ac93d4041f5 in term 0.
I20260925 15:22:59.415104 18411 leader_election.cc:302] T c30983c7ec7c4af6be799d0cf985cc6c P 0ebeb29ff8794c69802c8ac93d4041f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebeb29ff8794c69802c8ac93d4041f5, aa9b764a299e4504aab87753ddfab3c5; no voters:
I20260925 15:22:59.415287 18667 raft_consensus.cc:2781] T c30983c7ec7c4af6be799d0cf985cc6c P 0ebeb29ff8794c69802c8ac93d4041f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:22:59.415326 18667 raft_consensus.cc:482] T c30983c7ec7c4af6be799d0cf985cc6c P 0ebeb29ff8794c69802c8ac93d4041f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:22:59.415341 18667 raft_consensus.cc:3037] T c30983c7ec7c4af6be799d0cf985cc6c P 0ebeb29ff8794c69802c8ac93d4041f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:22:59.416076 18667 raft_consensus.cc:504] T c30983c7ec7c4af6be799d0cf985cc6c P 0ebeb29ff8794c69802c8ac93d4041f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebeb29ff8794c69802c8ac93d4041f5" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40125 } } peers { permanent_uuid: "aa9b764a299e4504aab87753ddfab3c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 45859 } } peers { permanent_uuid: "f06bc4bc858c491b893baed833d7e1cd" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34875 } }
I20260925 15:22:59.416229 18667 leader_election.cc:288] T c30983c7ec7c4af6be799d0cf985cc6c P 0ebeb29ff8794c69802c8ac93d4041f5 [CANDIDATE]: Term 1 election: Requested vote from peers aa9b764a299e4504aab87753ddfab3c5 (127.3.211.195:45859), f06bc4bc858c491b893baed833d7e1cd (127.3.211.193:34875)
I20260925 15:22:59.416415 18332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c30983c7ec7c4af6be799d0cf985cc6c" candidate_uuid: "0ebeb29ff8794c69802c8ac93d4041f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f06bc4bc858c491b893baed833d7e1cd"
I20260925 15:22:59.416440 18607 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c30983c7ec7c4af6be799d0cf985cc6c" candidate_uuid: "0ebeb29ff8794c69802c8ac93d4041f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa9b764a299e4504aab87753ddfab3c5"
I20260925 15:22:59.416514 18332 raft_consensus.cc:3037] T c30983c7ec7c4af6be799d0cf985cc6c P f06bc4bc858c491b893baed833d7e1cd [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:22:59.416534 18607 raft_consensus.cc:3037] T c30983c7ec7c4af6be799d0cf985cc6c P aa9b764a299e4504aab87753ddfab3c5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:22:59.417213 18607 raft_consensus.cc:2454] T c30983c7ec7c4af6be799d0cf985cc6c P aa9b764a299e4504aab87753ddfab3c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebeb29ff8794c69802c8ac93d4041f5 in term 1.
I20260925 15:22:59.417251 18332 raft_consensus.cc:2454] T c30983c7ec7c4af6be799d0cf985cc6c P f06bc4bc858c491b893baed833d7e1cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebeb29ff8794c69802c8ac93d4041f5 in term 1.
I20260925 15:22:59.417431 18411 leader_election.cc:302] T c30983c7ec7c4af6be799d0cf985cc6c P 0ebeb29ff8794c69802c8ac93d4041f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebeb29ff8794c69802c8ac93d4041f5, aa9b764a299e4504aab87753ddfab3c5; no voters:
I20260925 15:22:59.417544 18667 raft_consensus.cc:2781] T c30983c7ec7c4af6be799d0cf985cc6c P 0ebeb29ff8794c69802c8ac93d4041f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:22:59.417711 18667 raft_consensus.cc:686] T c30983c7ec7c4af6be799d0cf985cc6c P 0ebeb29ff8794c69802c8ac93d4041f5 [term 1 LEADER]: Becoming Leader. State: Replica: 0ebeb29ff8794c69802c8ac93d4041f5, State: Running, Role: LEADER
I20260925 15:22:59.417816 18667 consensus_queue.cc:237] T c30983c7ec7c4af6be799d0cf985cc6c P 0ebeb29ff8794c69802c8ac93d4041f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebeb29ff8794c69802c8ac93d4041f5" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40125 } } peers { permanent_uuid: "aa9b764a299e4504aab87753ddfab3c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 45859 } } peers { permanent_uuid: "f06bc4bc858c491b893baed833d7e1cd" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34875 } }
I20260925 15:22:59.418764 18194 catalog_manager.cc:5495] T c30983c7ec7c4af6be799d0cf985cc6c P 0ebeb29ff8794c69802c8ac93d4041f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ebeb29ff8794c69802c8ac93d4041f5 (127.3.211.194). New cstate: current_term: 1 leader_uuid: "0ebeb29ff8794c69802c8ac93d4041f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebeb29ff8794c69802c8ac93d4041f5" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa9b764a299e4504aab87753ddfab3c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 45859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f06bc4bc858c491b893baed833d7e1cd" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34875 } health_report { overall_health: UNKNOWN } } }
I20260925 15:22:59.433131 18659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:59.430636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50238 (local address 127.3.211.194:40125)
0925 15:22:59.430708 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 15:22:59.430710 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:22:59.430730 (+ 20us) server_negotiation.cc:407] Connection header received
0925 15:22:59.430778 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:22:59.430779 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:22:59.430797 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:22:59.430873 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:22:59.430993 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:59.431473 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:59.432230 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:22:59.432475 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:22:59.432560 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:22:59.432743 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:22:59.432858 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:22:59.432961 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 15:22:59.432980 (+ 19us) server_negotiation.cc:299] Negotiation successful
0925 15:22:59.433003 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 15:22:59.435141 18607 raft_consensus.cc:1261] T c30983c7ec7c4af6be799d0cf985cc6c P aa9b764a299e4504aab87753ddfab3c5 [term 1 FOLLOWER]: Refusing update from remote peer 0ebeb29ff8794c69802c8ac93d4041f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:22:59.435201 18332 raft_consensus.cc:1261] T c30983c7ec7c4af6be799d0cf985cc6c P f06bc4bc858c491b893baed833d7e1cd [term 1 FOLLOWER]: Refusing update from remote peer 0ebeb29ff8794c69802c8ac93d4041f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:22:59.435521 18667 consensus_queue.cc:1035] T c30983c7ec7c4af6be799d0cf985cc6c P 0ebeb29ff8794c69802c8ac93d4041f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f06bc4bc858c491b893baed833d7e1cd" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:22:59.435701 18670 consensus_queue.cc:1035] T c30983c7ec7c4af6be799d0cf985cc6c P 0ebeb29ff8794c69802c8ac93d4041f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa9b764a299e4504aab87753ddfab3c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 45859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:22:59.437964 18673 mvcc.cc:205] Tried to move back new op lower bound from 7333272696563122176 to 7333272696495919104. Current Snapshot: MvccSnapshot[applied={T|T < 7333272696563122176}]
I20260925 15:22:59.439265 18674 mvcc.cc:205] Tried to move back new op lower bound from 7333272696563122176 to 7333272696495919104. Current Snapshot: MvccSnapshot[applied={T|T < 7333272696563122176 or (T in {7333272696563122176})}]
I20260925 15:22:59.440642 18194 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:39474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:22:59.440743 18194 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:39474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:22:59.441552 18194 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68: Sending DeleteTablet for 3 replicas of tablet c30983c7ec7c4af6be799d0cf985cc6c
I20260925 15:22:59.441879 18317 tablet_service.cc:1514] Processing DeleteTablet for tablet c30983c7ec7c4af6be799d0cf985cc6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:22:59 UTC) from {username='kudu', principal='kudu/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:43350
I20260925 15:22:59.441882 18587 tablet_service.cc:1514] Processing DeleteTablet for tablet c30983c7ec7c4af6be799d0cf985cc6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:22:59 UTC) from {username='kudu', principal='kudu/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:52252
I20260925 15:22:59.441995 18392 tablet_replica.cc:330] T c30983c7ec7c4af6be799d0cf985cc6c P f06bc4bc858c491b893baed833d7e1cd: stopping tablet replica
I20260925 15:22:59.442049 18392 raft_consensus.cc:2229] T c30983c7ec7c4af6be799d0cf985cc6c P f06bc4bc858c491b893baed833d7e1cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:22:59.442090 18392 raft_consensus.cc:2258] T c30983c7ec7c4af6be799d0cf985cc6c P f06bc4bc858c491b893baed833d7e1cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:22:59.442106 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 18233
I20260925 15:22:59.442117 18455 tablet_service.cc:1514] Processing DeleteTablet for tablet c30983c7ec7c4af6be799d0cf985cc6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:22:59 UTC) from {username='kudu', principal='kudu/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:50230
I20260925 15:22:59.442389 18392 ts_tablet_manager.cc:1905] T c30983c7ec7c4af6be799d0cf985cc6c P f06bc4bc858c491b893baed833d7e1cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:22:59.442538 18683 tablet_replica.cc:330] T c30983c7ec7c4af6be799d0cf985cc6c P aa9b764a299e4504aab87753ddfab3c5: stopping tablet replica
I20260925 15:22:59.442837 18683 raft_consensus.cc:2229] T c30983c7ec7c4af6be799d0cf985cc6c P aa9b764a299e4504aab87753ddfab3c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:22:59.443017 18683 raft_consensus.cc:2258] T c30983c7ec7c4af6be799d0cf985cc6c P aa9b764a299e4504aab87753ddfab3c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:22:59.443583 18683 ts_tablet_manager.cc:1905] T c30983c7ec7c4af6be799d0cf985cc6c P aa9b764a299e4504aab87753ddfab3c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:22:59.445525 18683 ts_tablet_manager.cc:1918] T c30983c7ec7c4af6be799d0cf985cc6c P aa9b764a299e4504aab87753ddfab3c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:22:59.445618 18683 log.cc:1196] T c30983c7ec7c4af6be799d0cf985cc6c P aa9b764a299e4504aab87753ddfab3c5: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-2/wal/wals/c30983c7ec7c4af6be799d0cf985cc6c
I20260925 15:22:59.445962 18683 ts_tablet_manager.cc:1939] T c30983c7ec7c4af6be799d0cf985cc6c P aa9b764a299e4504aab87753ddfab3c5: Deleting consensus metadata
I20260925 15:22:59.446506 18182 catalog_manager.cc:4841] TS aa9b764a299e4504aab87753ddfab3c5 (127.3.211.195:45859): tablet c30983c7ec7c4af6be799d0cf985cc6c (table test-table [id=952db465628b4a67b12983aebd74fdf3]) successfully deleted
W20260925 15:22:59.448271 18181 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.211.193:34875 (error 108)
I20260925 15:22:59.448560 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 18388
I20260925 15:22:59.450367 18661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:59.450187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:34875 (local address 127.0.0.1:43356)
0925 15:22:59.450278 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:59.450325 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.193:34875: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260925 15:22:59.450507 18181 catalog_manager.cc:4584] TS f06bc4bc858c491b893baed833d7e1cd (127.3.211.193:34875): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c30983c7ec7c4af6be799d0cf985cc6c: Network error: Client connection negotiation failed: client connection to 127.3.211.193:34875: connect: Connection refused (error 111)
I20260925 15:22:59.454224 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 18524
I20260925 15:22:59.454366 18661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:59.454205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.194:40125 (local address 127.0.0.1:50244)
0925 15:22:59.454279 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:59.454317 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.194:40125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 15:22:59.460045 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 18162
I20260925 15:22:59.465824 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:34373
--webserver_interface=127.3.211.254
--webserver_port=40273
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:46329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.211.254:34373
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:22:59.551276 18686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:22:59.551549 18686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:22:59.551591 18686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:22:59.552839 18686 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:22:59.552917 18686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:22:59.552942 18686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:22:59.552970 18686 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:22:59.553000 18686 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:22:59.554518 18686 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:46329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.211.254:34373
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:34373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.211.254
--webserver_port=40273
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:22:59.554813 18686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:22:59.555052 18686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:22:59.560202 18691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:59.560204 18692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:22:59.560246 18686 server_base.cc:1034] running on GCE node
W20260925 15:22:59.560191 18694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:22:59.560623 18686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:22:59.560925 18686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:22:59.562108 18686 hybrid_clock.cc:648] HybridClock initialized: now 1790349779562114 us; error 48 us; skew 500 ppm
Sep 25 15:22:59 dist-test-slave-4bk3 krb5kdc[17614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349779, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:22:59.564796 18686 init.cc:377] Logged in from keytab as kudu/127.3.211.254@KRBTEST.COM (short username kudu)
I20260925 15:22:59.565310 18686 webserver.cc:460] Webserver started at http://127.3.211.254:40273/ using document root <none> and password file <none>
I20260925 15:22:59.565522 18686 fs_manager.cc:359] Metadata directory not provided
I20260925 15:22:59.565567 18686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:22:59.566898 18686 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:22:59.567570 18701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:59.567802 18686 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:22:59.567912 18686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/master-0/wal
uuid: "d18c834f3f6e4638a02dcc3eed10aa68"
format_stamp: "Formatted at 2026-09-25 15:22:57 on dist-test-slave-4bk3"
server_key: "74c164be27e4b8b724c41986d65353ee"
server_key_iv: "726dbd089ad157cbb45e55c653fa7e52"
server_key_version: "encryptionkey@0"
I20260925 15:22:59.568323 18686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:22:59.578547 18686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:22:59.580869 18686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:22:59.581029 18686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:22:59.585357 18686 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.254:34373
I20260925 15:22:59.585391 18753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.254:34373 every 8 connection(s)
I20260925 15:22:59.585812 18686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
I20260925 15:22:59.587433 18754 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:22:59.588043 18754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68: Bootstrap starting.
I20260925 15:22:59.591672 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 18686
I20260925 15:22:59.592010 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.193:34875
--local_ip_for_outbound_sockets=127.3.211.193
--tserver_master_addrs=127.3.211.254:34373
--webserver_port=36791
--webserver_interface=127.3.211.193
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab
--principal=kudu/127.3.211.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:46329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:22:59.596962 18754 log.cc:824] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68: Log is configured to *not* fsync() on all Append() calls
I20260925 15:22:59.599331 18754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:22:59.599687 18754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68: Bootstrap complete.
I20260925 15:22:59.602044 18754 raft_consensus.cc:348] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d18c834f3f6e4638a02dcc3eed10aa68" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 34373 } }
I20260925 15:22:59.602327 18754 raft_consensus.cc:729] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d18c834f3f6e4638a02dcc3eed10aa68, State: Initialized, Role: FOLLOWER
I20260925 15:22:59.602519 18754 consensus_queue.cc:260] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d18c834f3f6e4638a02dcc3eed10aa68" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 34373 } }
I20260925 15:22:59.602599 18754 raft_consensus.cc:388] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:22:59.602653 18754 raft_consensus.cc:482] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:22:59.602706 18754 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68 [term 2 FOLLOWER]: Advancing to term 3
I20260925 15:22:59.603762 18754 raft_consensus.cc:504] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d18c834f3f6e4638a02dcc3eed10aa68" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 34373 } }
I20260925 15:22:59.603891 18754 leader_election.cc:302] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d18c834f3f6e4638a02dcc3eed10aa68; no voters:
I20260925 15:22:59.604111 18754 leader_election.cc:288] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 15:22:59.604140 18758 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 15:22:59.604259 18758 raft_consensus.cc:686] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68 [term 3 LEADER]: Becoming Leader. State: Replica: d18c834f3f6e4638a02dcc3eed10aa68, State: Running, Role: LEADER
I20260925 15:22:59.604429 18754 sys_catalog.cc:564] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:22:59.604456 18758 consensus_queue.cc:237] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d18c834f3f6e4638a02dcc3eed10aa68" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 34373 } }
I20260925 15:22:59.604862 18760 sys_catalog.cc:455] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d18c834f3f6e4638a02dcc3eed10aa68. Latest consensus state: current_term: 3 leader_uuid: "d18c834f3f6e4638a02dcc3eed10aa68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d18c834f3f6e4638a02dcc3eed10aa68" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 34373 } } }
I20260925 15:22:59.604955 18760 sys_catalog.cc:458] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68 [sys.catalog]: This master's current role is: LEADER
I20260925 15:22:59.605189 18759 sys_catalog.cc:455] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d18c834f3f6e4638a02dcc3eed10aa68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d18c834f3f6e4638a02dcc3eed10aa68" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 34373 } } }
I20260925 15:22:59.605271 18759 sys_catalog.cc:458] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68 [sys.catalog]: This master's current role is: LEADER
I20260925 15:22:59.606424 18774 catalog_manager.cc:1259] Loaded cluster ID: 331dfeb1e768466396a7dee70f75b8e6
I20260925 15:22:59.606478 18774 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68: loading cluster ID for follower catalog manager: success
W20260925 15:22:59.606810 18774 catalog_manager.cc:1576] T 00000000000000000000000000000000 P d18c834f3f6e4638a02dcc3eed10aa68: acquiring CA information for follower catalog manager: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
W20260925 15:22:59.606861 18774 catalog_manager.cc:873] Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO: failed to prepare follower catalog manager, will retry
I20260925 15:22:59.606895 18765 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:22:59.607290 18765 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:22:59.607379 18765 catalog_manager.cc:1259] Loaded cluster ID: 331dfeb1e768466396a7dee70f75b8e6
I20260925 15:22:59.607412 18765 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 15:22:59.607535 18765 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f24282439ad google::LogMessage::Fail() at ??:0
@ 0x7f242824688a google::LogMessage::SendToLog() at ??:0
@ 0x7f24282434de google::LogMessage::Flush() at ??:0
@ 0x7f2428244a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f24301640f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f243016d8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f242a1110d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f242a106954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f242af6a6db start_thread at ??:0
@ 0x7f2426d4161f clone at ??:0
W20260925 15:22:59.695544 18756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:22:59.695784 18756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:22:59.695813 18756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:22:59.697257 18756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:22:59.697309 18756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:22:59.697340 18756 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.193
I20260925 15:22:59.698999 18756 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:46329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab
--principal=kudu/127.3.211.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.193:34875
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.211.193
--webserver_port=36791
--tserver_master_addrs=127.3.211.254:34373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18756
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.193
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:22:59.699288 18756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:22:59.699560 18756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:22:59.705307 18790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:59.705274 18791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:59.705271 18793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:22:59.706034 18756 server_base.cc:1034] running on GCE node
I20260925 15:22:59.706295 18756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:22:59.706609 18756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:22:59.707808 18756 hybrid_clock.cc:648] HybridClock initialized: now 1790349779707792 us; error 31 us; skew 500 ppm
Sep 25 15:22:59 dist-test-slave-4bk3 krb5kdc[17614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349779, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:22:59.710840 18756 init.cc:377] Logged in from keytab as kudu/127.3.211.193@KRBTEST.COM (short username kudu)
I20260925 15:22:59.711386 18756 webserver.cc:460] Webserver started at http://127.3.211.193:36791/ using document root <none> and password file <none>
I20260925 15:22:59.711635 18756 fs_manager.cc:359] Metadata directory not provided
I20260925 15:22:59.711710 18756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:22:59.713258 18756 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:22:59.714068 18800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:59.714335 18756 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:22:59.714443 18756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-0/wal
uuid: "f06bc4bc858c491b893baed833d7e1cd"
format_stamp: "Formatted at 2026-09-25 15:22:58 on dist-test-slave-4bk3"
server_key: "fd1d7454747e3eac10ae4b0f703fa59a"
server_key_iv: "d9b085bd2ead1a20cb2e76205e491cd0"
server_key_version: "encryptionkey@0"
I20260925 15:22:59.714954 18756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:22:59.738425 18756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:22:59.741139 18756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:22:59.741307 18756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:22:59.741576 18756 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:22:59.742102 18807 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 15:22:59.743728 18756 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 15:22:59.743799 18756 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 15:22:59.743839 18756 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 15:22:59.744802 18756 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 15:22:59.744853 18756 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.001s
I20260925 15:22:59.744995 18807 tablet_bootstrap.cc:492] T c30983c7ec7c4af6be799d0cf985cc6c P f06bc4bc858c491b893baed833d7e1cd: Bootstrap starting.
I20260925 15:22:59.746728 18807 log.cc:824] T c30983c7ec7c4af6be799d0cf985cc6c P f06bc4bc858c491b893baed833d7e1cd: Log is configured to *not* fsync() on all Append() calls
I20260925 15:22:59.750183 18807 tablet_bootstrap.cc:492] T c30983c7ec7c4af6be799d0cf985cc6c P f06bc4bc858c491b893baed833d7e1cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:22:59.750550 18807 tablet_bootstrap.cc:492] T c30983c7ec7c4af6be799d0cf985cc6c P f06bc4bc858c491b893baed833d7e1cd: Bootstrap complete.
I20260925 15:22:59.750728 18807 ts_tablet_manager.cc:1397] T c30983c7ec7c4af6be799d0cf985cc6c P f06bc4bc858c491b893baed833d7e1cd: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260925 15:22:59.752128 18756 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.193:34875
I20260925 15:22:59.752182 18915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.193:34875 every 8 connection(s)
I20260925 15:22:59.752629 18756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:22:59.752815 18807 raft_consensus.cc:348] T c30983c7ec7c4af6be799d0cf985cc6c P f06bc4bc858c491b893baed833d7e1cd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebeb29ff8794c69802c8ac93d4041f5" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40125 } } peers { permanent_uuid: "aa9b764a299e4504aab87753ddfab3c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 45859 } } peers { permanent_uuid: "f06bc4bc858c491b893baed833d7e1cd" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34875 } }
I20260925 15:22:59.753618 18807 raft_consensus.cc:729] T c30983c7ec7c4af6be799d0cf985cc6c P f06bc4bc858c491b893baed833d7e1cd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f06bc4bc858c491b893baed833d7e1cd, State: Initialized, Role: FOLLOWER
I20260925 15:22:59.753767 18807 consensus_queue.cc:260] T c30983c7ec7c4af6be799d0cf985cc6c P f06bc4bc858c491b893baed833d7e1cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebeb29ff8794c69802c8ac93d4041f5" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40125 } } peers { permanent_uuid: "aa9b764a299e4504aab87753ddfab3c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 45859 } } peers { permanent_uuid: "f06bc4bc858c491b893baed833d7e1cd" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34875 } }
I20260925 15:22:59.754016 18807 ts_tablet_manager.cc:1428] T c30983c7ec7c4af6be799d0cf985cc6c P f06bc4bc858c491b893baed833d7e1cd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:22:59.757954 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 18756
I20260925 15:22:59.758288 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.194:40125
--local_ip_for_outbound_sockets=127.3.211.194
--tserver_master_addrs=127.3.211.254:34373
--webserver_port=36343
--webserver_interface=127.3.211.194
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab
--principal=kudu/127.3.211.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:46329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:22:59.836934 18918 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:22:59.753314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.193:53151)
0925 15:22:59.753488 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:59.753554 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 15:22:59.753719 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:22:59.836728 (+ 83009us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":90,"threads_started":1}
W20260925 15:22:59.837231 18916 heartbeater.cc:644] Failed to heartbeat to 127.3.211.254:34373 (0 consecutive failures): Network error: Failed to ping master at 127.3.211.254:34373: Client connection negotiation failed: client connection to 127.3.211.254:34373: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 15:22:59.837657 18918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:59.837491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.193:55569)
0925 15:22:59.837558 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:59.837619 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 15:22:59.838007 18918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:59.837890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.193:55103)
0925 15:22:59.837947 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:59.837979 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 15:22:59.838171 18916 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 15:22:59.849040 18920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:22:59.849268 18920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:22:59.849298 18920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:22:59.850600 18920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:22:59.850662 18920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:22:59.850695 18920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.194
I20260925 15:22:59.852312 18920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:46329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab
--principal=kudu/127.3.211.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.194:40125
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.211.194
--webserver_port=36343
--tserver_master_addrs=127.3.211.254:34373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.194
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:22:59.852669 18920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:22:59.852936 18920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 15:22:59.859110 18920 server_base.cc:1034] running on GCE node
W20260925 15:22:59.858974 18928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:59.858974 18925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:59.858979 18926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:22:59.859591 18920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:22:59.859889 18920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:22:59.861069 18920 hybrid_clock.cc:648] HybridClock initialized: now 1790349779861050 us; error 48 us; skew 500 ppm
Sep 25 15:22:59 dist-test-slave-4bk3 krb5kdc[17614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349779, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:22:59.863740 18920 init.cc:377] Logged in from keytab as kudu/127.3.211.194@KRBTEST.COM (short username kudu)
I20260925 15:22:59.864181 18920 webserver.cc:460] Webserver started at http://127.3.211.194:36343/ using document root <none> and password file <none>
I20260925 15:22:59.864459 18920 fs_manager.cc:359] Metadata directory not provided
I20260925 15:22:59.864511 18920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:22:59.865952 18920 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:22:59.866758 18935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:22:59.867086 18920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:22:59.867169 18920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-1/wal
uuid: "0ebeb29ff8794c69802c8ac93d4041f5"
format_stamp: "Formatted at 2026-09-25 15:22:58 on dist-test-slave-4bk3"
server_key: "f3d329eae3527319e15752293c4cf17d"
server_key_iv: "24d7c8213763f7105ea359c64b904acf"
server_key_version: "encryptionkey@0"
I20260925 15:22:59.867535 18920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:22:59.882225 18920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:22:59.884830 18920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:22:59.884960 18920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:22:59.885160 18920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:22:59.885619 18942 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 15:22:59.887087 18920 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 15:22:59.887151 18920 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 15:22:59.887178 18920 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 15:22:59.887899 18920 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 15:22:59.887938 18920 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 15:22:59.888073 18942 tablet_bootstrap.cc:492] T c30983c7ec7c4af6be799d0cf985cc6c P 0ebeb29ff8794c69802c8ac93d4041f5: Bootstrap starting.
I20260925 15:22:59.894901 18920 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.194:40125
I20260925 15:22:59.895123 19049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.194:40125 every 8 connection(s)
I20260925 15:22:59.895370 18920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
I20260925 15:22:59.896615 19052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:59.896048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.194:48791)
0925 15:22:59.896245 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:59.896392 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":78,"threads_started":1}
W20260925 15:22:59.896837 19050 heartbeater.cc:644] Failed to heartbeat to 127.3.211.254:34373 (0 consecutive failures): Network error: Failed to ping master at 127.3.211.254:34373: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
I20260925 15:22:59.897186 19052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:59.897071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.194:47247)
0925 15:22:59.897132 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:59.897163 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 15:22:59.897497 19052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:59.897437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.194:42641)
0925 15:22:59.897465 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:59.897484 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260925 15:22:59.897620 19050 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 15:22:59.899508 18942 log.cc:824] T c30983c7ec7c4af6be799d0cf985cc6c P 0ebeb29ff8794c69802c8ac93d4041f5: Log is configured to *not* fsync() on all Append() calls
I20260925 15:22:59.901929 18942 tablet_bootstrap.cc:492] T c30983c7ec7c4af6be799d0cf985cc6c P 0ebeb29ff8794c69802c8ac93d4041f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:22:59.902274 18942 tablet_bootstrap.cc:492] T c30983c7ec7c4af6be799d0cf985cc6c P 0ebeb29ff8794c69802c8ac93d4041f5: Bootstrap complete.
I20260925 15:22:59.903266 18942 ts_tablet_manager.cc:1397] T c30983c7ec7c4af6be799d0cf985cc6c P 0ebeb29ff8794c69802c8ac93d4041f5: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.009s
I20260925 15:22:59.904568 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 18920
I20260925 15:22:59.904798 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.195:45859
--local_ip_for_outbound_sockets=127.3.211.195
--tserver_master_addrs=127.3.211.254:34373
--webserver_port=41357
--webserver_interface=127.3.211.195
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab
--principal=kudu/127.3.211.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:46329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:22:59.904837 18942 raft_consensus.cc:348] T c30983c7ec7c4af6be799d0cf985cc6c P 0ebeb29ff8794c69802c8ac93d4041f5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebeb29ff8794c69802c8ac93d4041f5" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40125 } } peers { permanent_uuid: "aa9b764a299e4504aab87753ddfab3c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 45859 } } peers { permanent_uuid: "f06bc4bc858c491b893baed833d7e1cd" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34875 } }
I20260925 15:22:59.905261 18942 raft_consensus.cc:729] T c30983c7ec7c4af6be799d0cf985cc6c P 0ebeb29ff8794c69802c8ac93d4041f5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ebeb29ff8794c69802c8ac93d4041f5, State: Initialized, Role: FOLLOWER
I20260925 15:22:59.905435 18942 consensus_queue.cc:260] T c30983c7ec7c4af6be799d0cf985cc6c P 0ebeb29ff8794c69802c8ac93d4041f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebeb29ff8794c69802c8ac93d4041f5" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40125 } } peers { permanent_uuid: "aa9b764a299e4504aab87753ddfab3c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 45859 } } peers { permanent_uuid: "f06bc4bc858c491b893baed833d7e1cd" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34875 } }
I20260925 15:22:59.905804 18942 ts_tablet_manager.cc:1428] T c30983c7ec7c4af6be799d0cf985cc6c P 0ebeb29ff8794c69802c8ac93d4041f5: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20260925 15:22:59.906144 19052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:59.906004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.194:49071)
0925 15:22:59.906073 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:59.906108 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 15:22:59.906497 19052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:22:59.906405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.194:54579)
0925 15:22:59.906451 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 15:22:59.906476 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 15:22:59.988456 19055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:22:59.988694 19055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:22:59.988725 19055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:22:59.990006 19055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:22:59.990061 19055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:22:59.990092 19055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.195
I20260925 15:22:59.991700 19055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:46329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab
--principal=kudu/127.3.211.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.195:45859
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.211.195
--webserver_port=41357
--tserver_master_addrs=127.3.211.254:34373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.195
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:22:59.992002 19055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:22:59.992261 19055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:22:59.997632 19060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:22:59.997787 19055 server_base.cc:1034] running on GCE node
W20260925 15:22:59.997632 19063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:22:59.997654 19061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:22:59.998128 19055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:22:59.998425 19055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:22:59.999611 19055 hybrid_clock.cc:648] HybridClock initialized: now 1790349779999590 us; error 40 us; skew 500 ppm
Sep 25 15:22:59 dist-test-slave-4bk3 krb5kdc[17614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349779, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:23:00.002429 19055 init.cc:377] Logged in from keytab as kudu/127.3.211.195@KRBTEST.COM (short username kudu)
I20260925 15:23:00.002969 19055 webserver.cc:460] Webserver started at http://127.3.211.195:41357/ using document root <none> and password file <none>
I20260925 15:23:00.003214 19055 fs_manager.cc:359] Metadata directory not provided
I20260925 15:23:00.003288 19055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:23:00.004945 19055 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:23:00.006181 19070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:00.006496 19055 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:23:00.006639 19055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-2/wal
uuid: "aa9b764a299e4504aab87753ddfab3c5"
format_stamp: "Formatted at 2026-09-25 15:22:58 on dist-test-slave-4bk3"
server_key: "73509fac2ace0aa24b5499535efcf550"
server_key_iv: "886bf768a689649c7627c90016fd9a0f"
server_key_version: "encryptionkey@0"
I20260925 15:23:00.007053 19055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:23:00.018620 19055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:23:00.021179 19055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:23:00.021377 19055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:23:00.021641 19055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:23:00.021978 19055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:23:00.022046 19055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:00.022082 19055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:23:00.022127 19055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:00.028189 19055 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.195:45859
I20260925 15:23:00.028230 19183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.195:45859 every 8 connection(s)
I20260925 15:23:00.028641 19055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
I20260925 15:23:00.029794 19186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:00.029375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.195:42583)
0925 15:23:00.029570 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:00.029649 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":76,"threads_started":1}
W20260925 15:23:00.030269 19184 heartbeater.cc:644] Failed to heartbeat to 127.3.211.254:34373 (0 consecutive failures): Network error: Failed to ping master at 127.3.211.254:34373: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
I20260925 15:23:00.030634 19186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:00.030508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.195:35571)
0925 15:23:00.030569 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:00.030611 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 15:23:00.030632 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 19055
I20260925 15:23:00.030992 19186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:00.030839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.195:36595)
0925 15:23:00.030938 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:00.030970 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260925 15:23:00.031220 19184 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 15:23:00.838883 19188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:00.838607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.193:50279)
0925 15:23:00.838759 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:00.838813 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":66,"threads_started":1}
I20260925 15:23:00.907384 19189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:00.907093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.194:33949)
0925 15:23:00.907258 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:00.907320 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":69,"threads_started":1}
I20260925 15:23:01.031925 19190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:01.031683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.195:37525)
0925 15:23:01.031828 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:01.031867 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":64,"threads_started":1}
I20260925 15:23:01.155405 19191 raft_consensus.cc:482] T c30983c7ec7c4af6be799d0cf985cc6c P f06bc4bc858c491b893baed833d7e1cd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:23:01.155610 19191 raft_consensus.cc:504] T c30983c7ec7c4af6be799d0cf985cc6c P f06bc4bc858c491b893baed833d7e1cd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebeb29ff8794c69802c8ac93d4041f5" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40125 } } peers { permanent_uuid: "aa9b764a299e4504aab87753ddfab3c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 45859 } } peers { permanent_uuid: "f06bc4bc858c491b893baed833d7e1cd" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34875 } }
I20260925 15:23:01.155992 19191 leader_election.cc:288] T c30983c7ec7c4af6be799d0cf985cc6c P f06bc4bc858c491b893baed833d7e1cd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0ebeb29ff8794c69802c8ac93d4041f5 (127.3.211.194:40125), aa9b764a299e4504aab87753ddfab3c5 (127.3.211.195:45859)
Sep 25 15:23:01 dist-test-slave-4bk3 krb5kdc[17614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349779, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.193@KRBTEST.COM for kudu/127.3.211.195@KRBTEST.COM
Sep 25 15:23:01 dist-test-slave-4bk3 krb5kdc[17614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349779, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.193@KRBTEST.COM for kudu/127.3.211.194@KRBTEST.COM
I20260925 15:23:01.169142 19192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:01.156308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.195:45859 (local address 127.3.211.193:34037)
0925 15:23:01.156467 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:01.156475 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:01.156515 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:01.157269 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:01.157269 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:01.157500 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:23:01.157827 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:01.157838 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:01.159728 (+ 1890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:01.159730 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:01.161102 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:01.161105 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:01.162002 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:01.162003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:01.162183 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:01.162926 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:23:01.162937 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:23:01.164866 (+ 1929us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:23:01.167239 (+ 2373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:23:01.167251 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:23:01.167254 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:23:01.167614 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:23:01.167907 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:23:01.167908 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:23:01.167908 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:23:01.168054 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:23:01.168549 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:23:01.168556 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:23:01.168714 (+ 158us) client_negotiation.cc:769] Sending connection context
0925 15:23:01.168818 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 15:23:01.169011 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":168,"thread_start_us":58,"threads_started":1}
I20260925 15:23:01.169162 19194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:01.156482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:34037 (local address 127.3.211.195:45859)
0925 15:23:01.156645 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:01.156654 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:01.156682 (+ 28us) server_negotiation.cc:407] Connection header received
0925 15:23:01.157110 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:01.157112 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:01.157149 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:01.157259 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:01.158002 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:01.159566 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:01.161230 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:01.161870 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:01.161995 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:01.165011 (+ 3016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:01.165018 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:01.165018 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:23:01.165035 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:01.167127 (+ 2092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:01.167722 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:01.167726 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:01.167726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:01.167832 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:01.168137 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:01.168138 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:01.168138 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:01.168483 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:01.168570 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:01.168898 (+ 328us) server_negotiation.cc:299] Negotiation successful
0925 15:23:01.169052 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":68,"threads_started":1}
I20260925 15:23:01.169291 19193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:01.156483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:57897 (local address 127.3.211.194:40125)
0925 15:23:01.156640 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:01.156650 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:01.156670 (+ 20us) server_negotiation.cc:407] Connection header received
0925 15:23:01.156791 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:01.156793 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:01.156833 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:01.156967 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:01.158145 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:01.159566 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:01.161230 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:01.161863 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:01.161996 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:01.165431 (+ 3435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:01.165435 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:01.165435 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:23:01.165448 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:01.167123 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:01.167716 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:01.167720 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:01.167720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:01.167832 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:01.168163 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:01.168163 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:01.168164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:01.168494 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:01.168978 (+ 484us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:01.169059 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 15:23:01.169189 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":56,"threads_started":1}
I20260925 15:23:01.169203 19188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:01.156307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.194:40125 (local address 127.3.211.193:57897)
0925 15:23:01.156395 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:01.156407 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:01.156467 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:01.157022 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:01.157024 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:01.157497 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:23:01.158028 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:01.158030 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:01.159743 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:01.159744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:01.161102 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:01.161105 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:01.162002 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:01.162003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:01.162183 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:01.162940 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:23:01.162944 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:23:01.165307 (+ 2363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:23:01.167270 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:23:01.167271 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:23:01.167272 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:23:01.167614 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:23:01.167907 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:23:01.167908 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:23:01.167908 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:23:01.168057 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:23:01.168589 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:23:01.168590 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:23:01.168714 (+ 124us) client_negotiation.cc:769] Sending connection context
0925 15:23:01.168808 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 15:23:01.169074 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"mutex_wait_us":172}
I20260925 15:23:01.170045 19004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c30983c7ec7c4af6be799d0cf985cc6c" candidate_uuid: "f06bc4bc858c491b893baed833d7e1cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "0ebeb29ff8794c69802c8ac93d4041f5" is_pre_election: true
I20260925 15:23:01.170215 19004 raft_consensus.cc:2454] T c30983c7ec7c4af6be799d0cf985cc6c P 0ebeb29ff8794c69802c8ac93d4041f5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f06bc4bc858c491b893baed833d7e1cd in term 1.
W20260925 15:23:01.170499 18804 leader_election.cc:341] T c30983c7ec7c4af6be799d0cf985cc6c P f06bc4bc858c491b893baed833d7e1cd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer aa9b764a299e4504aab87753ddfab3c5 (127.3.211.195:45859): Not found: Tablet not found: c30983c7ec7c4af6be799d0cf985cc6c
I20260925 15:23:01.170621 18803 leader_election.cc:302] T c30983c7ec7c4af6be799d0cf985cc6c P f06bc4bc858c491b893baed833d7e1cd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebeb29ff8794c69802c8ac93d4041f5, f06bc4bc858c491b893baed833d7e1cd; no voters: aa9b764a299e4504aab87753ddfab3c5
I20260925 15:23:01.170763 19191 raft_consensus.cc:2781] T c30983c7ec7c4af6be799d0cf985cc6c P f06bc4bc858c491b893baed833d7e1cd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 15:23:01.170825 19191 raft_consensus.cc:482] T c30983c7ec7c4af6be799d0cf985cc6c P f06bc4bc858c491b893baed833d7e1cd [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:23:01.170857 19191 raft_consensus.cc:3037] T c30983c7ec7c4af6be799d0cf985cc6c P f06bc4bc858c491b893baed833d7e1cd [term 1 FOLLOWER]: Advancing to term 2
I20260925 15:23:01.172133 19191 raft_consensus.cc:504] T c30983c7ec7c4af6be799d0cf985cc6c P f06bc4bc858c491b893baed833d7e1cd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebeb29ff8794c69802c8ac93d4041f5" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40125 } } peers { permanent_uuid: "aa9b764a299e4504aab87753ddfab3c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 45859 } } peers { permanent_uuid: "f06bc4bc858c491b893baed833d7e1cd" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34875 } }
I20260925 15:23:01.172290 19191 leader_election.cc:288] T c30983c7ec7c4af6be799d0cf985cc6c P f06bc4bc858c491b893baed833d7e1cd [CANDIDATE]: Term 2 election: Requested vote from peers 0ebeb29ff8794c69802c8ac93d4041f5 (127.3.211.194:40125), aa9b764a299e4504aab87753ddfab3c5 (127.3.211.195:45859)
I20260925 15:23:01.172510 19004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c30983c7ec7c4af6be799d0cf985cc6c" candidate_uuid: "f06bc4bc858c491b893baed833d7e1cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "0ebeb29ff8794c69802c8ac93d4041f5"
I20260925 15:23:01.172641 19004 raft_consensus.cc:3037] T c30983c7ec7c4af6be799d0cf985cc6c P 0ebeb29ff8794c69802c8ac93d4041f5 [term 1 FOLLOWER]: Advancing to term 2
W20260925 15:23:01.172665 18804 leader_election.cc:341] T c30983c7ec7c4af6be799d0cf985cc6c P f06bc4bc858c491b893baed833d7e1cd [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer aa9b764a299e4504aab87753ddfab3c5 (127.3.211.195:45859): Not found: Tablet not found: c30983c7ec7c4af6be799d0cf985cc6c
I20260925 15:23:01.173805 19004 raft_consensus.cc:2454] T c30983c7ec7c4af6be799d0cf985cc6c P 0ebeb29ff8794c69802c8ac93d4041f5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f06bc4bc858c491b893baed833d7e1cd in term 2.
I20260925 15:23:01.174015 18803 leader_election.cc:302] T c30983c7ec7c4af6be799d0cf985cc6c P f06bc4bc858c491b893baed833d7e1cd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ebeb29ff8794c69802c8ac93d4041f5, f06bc4bc858c491b893baed833d7e1cd; no voters: aa9b764a299e4504aab87753ddfab3c5
I20260925 15:23:01.174121 19191 raft_consensus.cc:2781] T c30983c7ec7c4af6be799d0cf985cc6c P f06bc4bc858c491b893baed833d7e1cd [term 2 FOLLOWER]: Leader election won for term 2
I20260925 15:23:01.174248 19191 raft_consensus.cc:686] T c30983c7ec7c4af6be799d0cf985cc6c P f06bc4bc858c491b893baed833d7e1cd [term 2 LEADER]: Becoming Leader. State: Replica: f06bc4bc858c491b893baed833d7e1cd, State: Running, Role: LEADER
I20260925 15:23:01.174345 19191 consensus_queue.cc:237] T c30983c7ec7c4af6be799d0cf985cc6c P f06bc4bc858c491b893baed833d7e1cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebeb29ff8794c69802c8ac93d4041f5" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40125 } } peers { permanent_uuid: "aa9b764a299e4504aab87753ddfab3c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 45859 } } peers { permanent_uuid: "f06bc4bc858c491b893baed833d7e1cd" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34875 } }
I20260925 15:23:01.174731 19188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:01.174556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.193:36453)
0925 15:23:01.174640 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:01.174680 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260925 15:23:01.582825 18804 consensus_peers.cc:487] T c30983c7ec7c4af6be799d0cf985cc6c P f06bc4bc858c491b893baed833d7e1cd -> Peer aa9b764a299e4504aab87753ddfab3c5 (127.3.211.195:45859): Couldn't send request to peer aa9b764a299e4504aab87753ddfab3c5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c30983c7ec7c4af6be799d0cf985cc6c. This is attempt 1: this message will repeat every 5th retry.
I20260925 15:23:01.678151 19004 raft_consensus.cc:1261] T c30983c7ec7c4af6be799d0cf985cc6c P 0ebeb29ff8794c69802c8ac93d4041f5 [term 2 FOLLOWER]: Refusing update from remote peer f06bc4bc858c491b893baed833d7e1cd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260925 15:23:01.678686 19191 consensus_queue.cc:1035] T c30983c7ec7c4af6be799d0cf985cc6c P f06bc4bc858c491b893baed833d7e1cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebeb29ff8794c69802c8ac93d4041f5" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260925 15:23:01.679901 19199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:01.679667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.194:33407)
0925 15:23:01.679770 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:01.679847 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":68,"threads_started":1}
I20260925 15:23:01.680840 19203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:01.680560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.193:37321)
0925 15:23:01.680736 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:01.680776 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":80,"threads_started":1}
I20260925 15:23:01.680990 19199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:01.680800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.194:35935)
0925 15:23:01.680879 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:01.680958 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 15:23:02.033061 19204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:02.032727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.195:49379)
0925 15:23:02.032948 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:02.033012 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":116,"threads_started":1}
I20260925 15:23:02.066805 19205 ts_tablet_manager.cc:927] T c30983c7ec7c4af6be799d0cf985cc6c P aa9b764a299e4504aab87753ddfab3c5: Initiating tablet copy from peer f06bc4bc858c491b893baed833d7e1cd (127.3.211.193:34875)
I20260925 15:23:02.067085 19205 tablet_copy_client.cc:316] T c30983c7ec7c4af6be799d0cf985cc6c P aa9b764a299e4504aab87753ddfab3c5: tablet copy: Beginning tablet copy session from remote peer at address 127.3.211.193:34875
Sep 25 15:23:02 dist-test-slave-4bk3 krb5kdc[17614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349779, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.195@KRBTEST.COM for kudu/127.3.211.193@KRBTEST.COM
I20260925 15:23:02.077392 19204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:02.067552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:34875 (local address 127.3.211.195:39179)
0925 15:23:02.067646 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:02.067690 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:02.067771 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:02.068127 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:02.068134 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:02.068548 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:23:02.068628 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:02.068631 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:02.069778 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:02.069780 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:02.070573 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:02.070577 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:02.071352 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:02.071354 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:02.071452 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:02.072005 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:23:02.072009 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:23:02.073756 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:23:02.075590 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:23:02.075599 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:23:02.075601 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:23:02.075820 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:23:02.076286 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:23:02.076287 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:23:02.076288 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:23:02.076563 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:23:02.077019 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:23:02.077028 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:23:02.077088 (+ 60us) client_negotiation.cc:769] Sending connection context
0925 15:23:02.077168 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 15:23:02.077296 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 15:23:02.077409 19207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:02.067737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:39179 (local address 127.3.211.193:34875)
0925 15:23:02.067893 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:02.067897 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:02.067911 (+ 14us) server_negotiation.cc:407] Connection header received
0925 15:23:02.067998 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:02.068001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:02.068015 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:02.068092 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:02.068749 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:02.069582 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:02.070877 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:02.071263 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:02.071331 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:02.073892 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:02.073901 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:02.073901 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:23:02.073910 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:02.075482 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:02.075995 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:02.076001 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:02.076002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:02.076144 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:02.076703 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:02.076705 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:02.076705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:02.076913 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:02.077013 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:02.077259 (+ 246us) server_negotiation.cc:299] Negotiation successful
0925 15:23:02.077338 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":71,"threads_started":1}
I20260925 15:23:02.078117 18890 tablet_copy_service.cc:140] P f06bc4bc858c491b893baed833d7e1cd: Received BeginTabletCopySession request for tablet c30983c7ec7c4af6be799d0cf985cc6c from peer aa9b764a299e4504aab87753ddfab3c5 ({username='kudu', principal='kudu/127.3.211.195@KRBTEST.COM'} at 127.3.211.195:39179)
I20260925 15:23:02.078382 18890 tablet_copy_service.cc:161] P f06bc4bc858c491b893baed833d7e1cd: Beginning new tablet copy session on tablet c30983c7ec7c4af6be799d0cf985cc6c from peer aa9b764a299e4504aab87753ddfab3c5 at {username='kudu', principal='kudu/127.3.211.195@KRBTEST.COM'} at 127.3.211.195:39179: session id = aa9b764a299e4504aab87753ddfab3c5-c30983c7ec7c4af6be799d0cf985cc6c
I20260925 15:23:02.078984 18890 tablet_copy_source_session.cc:205] T c30983c7ec7c4af6be799d0cf985cc6c P f06bc4bc858c491b893baed833d7e1cd: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 15:23:02.080148 19205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c30983c7ec7c4af6be799d0cf985cc6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:23:02.082775 19205 tablet_copy_client.cc:791] T c30983c7ec7c4af6be799d0cf985cc6c P aa9b764a299e4504aab87753ddfab3c5: tablet copy: Starting download of 0 data blocks...
I20260925 15:23:02.082935 19205 tablet_copy_client.cc:655] T c30983c7ec7c4af6be799d0cf985cc6c P aa9b764a299e4504aab87753ddfab3c5: tablet copy: Starting download of 1 WAL segments...
I20260925 15:23:02.084479 19205 tablet_copy_client.cc:531] T c30983c7ec7c4af6be799d0cf985cc6c P aa9b764a299e4504aab87753ddfab3c5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 15:23:02.085611 19205 tablet_bootstrap.cc:492] T c30983c7ec7c4af6be799d0cf985cc6c P aa9b764a299e4504aab87753ddfab3c5: Bootstrap starting.
I20260925 15:23:02.087455 19205 log.cc:824] T c30983c7ec7c4af6be799d0cf985cc6c P aa9b764a299e4504aab87753ddfab3c5: Log is configured to *not* fsync() on all Append() calls
I20260925 15:23:02.090994 19205 tablet_bootstrap.cc:492] T c30983c7ec7c4af6be799d0cf985cc6c P aa9b764a299e4504aab87753ddfab3c5: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:23:02.091298 19205 tablet_bootstrap.cc:492] T c30983c7ec7c4af6be799d0cf985cc6c P aa9b764a299e4504aab87753ddfab3c5: Bootstrap complete.
I20260925 15:23:02.091475 19205 ts_tablet_manager.cc:1397] T c30983c7ec7c4af6be799d0cf985cc6c P aa9b764a299e4504aab87753ddfab3c5: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260925 15:23:02.092641 19205 raft_consensus.cc:348] T c30983c7ec7c4af6be799d0cf985cc6c P aa9b764a299e4504aab87753ddfab3c5 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebeb29ff8794c69802c8ac93d4041f5" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40125 } } peers { permanent_uuid: "aa9b764a299e4504aab87753ddfab3c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 45859 } } peers { permanent_uuid: "f06bc4bc858c491b893baed833d7e1cd" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34875 } }
I20260925 15:23:02.092840 19205 raft_consensus.cc:729] T c30983c7ec7c4af6be799d0cf985cc6c P aa9b764a299e4504aab87753ddfab3c5 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa9b764a299e4504aab87753ddfab3c5, State: Initialized, Role: FOLLOWER
I20260925 15:23:02.093029 19205 consensus_queue.cc:260] T c30983c7ec7c4af6be799d0cf985cc6c P aa9b764a299e4504aab87753ddfab3c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ebeb29ff8794c69802c8ac93d4041f5" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40125 } } peers { permanent_uuid: "aa9b764a299e4504aab87753ddfab3c5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 45859 } } peers { permanent_uuid: "f06bc4bc858c491b893baed833d7e1cd" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 34875 } }
I20260925 15:23:02.093333 19205 ts_tablet_manager.cc:1428] T c30983c7ec7c4af6be799d0cf985cc6c P aa9b764a299e4504aab87753ddfab3c5: Time spent starting tablet: real 0.002s user 0.001s sys 0.004s
I20260925 15:23:02.093873 19204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:02.093738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.195:45031)
0925 15:23:02.093797 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:02.093831 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 15:23:02.093909 18890 tablet_copy_service.cc:342] P f06bc4bc858c491b893baed833d7e1cd: Request end of tablet copy session aa9b764a299e4504aab87753ddfab3c5-c30983c7ec7c4af6be799d0cf985cc6c received from {username='kudu', principal='kudu/127.3.211.195@KRBTEST.COM'} at 127.3.211.195:39179
I20260925 15:23:02.094002 18890 tablet_copy_service.cc:433] P f06bc4bc858c491b893baed833d7e1cd: ending tablet copy session aa9b764a299e4504aab87753ddfab3c5-c30983c7ec7c4af6be799d0cf985cc6c on tablet c30983c7ec7c4af6be799d0cf985cc6c with peer aa9b764a299e4504aab87753ddfab3c5
I20260925 15:23:02.488462 19212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:02.488218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.193:55511)
0925 15:23:02.488329 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:02.488405 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":49,"threads_started":1}
I20260925 15:23:02.488477 19138 raft_consensus.cc:1203] T c30983c7ec7c4af6be799d0cf985cc6c P aa9b764a299e4504aab87753ddfab3c5 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260925 15:23:02.488938 19204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:02.488831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.195:49607)
0925 15:23:02.488868 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:02.488904 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260925 15:23:02.681612 19213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:02.681397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.194:34947)
0925 15:23:02.681535 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:02.681567 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":80,"threads_started":1}
I20260925 15:23:03.489408 19214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:03.489085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.193:50979)
0925 15:23:03.489268 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:03.489314 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":70,"threads_started":1}
I20260925 15:23:03.489811 19215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:03.489514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.195:42239)
0925 15:23:03.489737 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:03.489774 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":116,"threads_started":1}
I20260925 15:23:03.682559 19216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:03.682278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.194:34645)
0925 15:23:03.682458 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:03.682510 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":78,"threads_started":1}
I20260925 15:23:04.490420 19217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:04.490145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.193:47039)
0925 15:23:04.490311 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:04.490362 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":148,"threads_started":1}
I20260925 15:23:04.490998 19218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:04.490688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.195:60299)
0925 15:23:04.490874 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:04.490939 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":74,"threads_started":1}
I20260925 15:23:04.683431 19219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:04.683181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.194:52957)
0925 15:23:04.683337 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:04.683384 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":77,"threads_started":1}
I20260925 15:23:05.491489 19220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:05.491234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.193:56583)
0925 15:23:05.491395 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:05.491441 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":69,"threads_started":1}
I20260925 15:23:05.491943 19221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:05.491682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.195:57557)
0925 15:23:05.491846 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:05.491883 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":95,"threads_started":1}
I20260925 15:23:05.491959 19220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:05.491858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.193:35553)
0925 15:23:05.491917 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:05.491942 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 15:23:05.684365 19222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:05.684083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.194:47161)
0925 15:23:05.684244 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:05.684292 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":65,"threads_started":1}
I20260925 15:23:06.492770 19223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:06.492534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.195:55147)
0925 15:23:06.492681 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:06.492717 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":88,"threads_started":1}
I20260925 15:23:06.493137 19224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:06.492579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.193:57311)
0925 15:23:06.492709 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:06.492757 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":76,"threads_started":1}
I20260925 15:23:06.685220 19225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:06.684953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.194:36057)
0925 15:23:06.685113 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:06.685172 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":62,"threads_started":1}
I20260925 15:23:07.493604 19226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:07.493344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.195:36527)
0925 15:23:07.493489 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:07.493557 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":64,"threads_started":1}
I20260925 15:23:07.493995 19227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:07.493809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.193:49997)
0925 15:23:07.493919 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:07.493958 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":70,"threads_started":1}
I20260925 15:23:07.686244 19228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:07.685969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.194:55271)
0925 15:23:07.686141 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:07.686195 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":77,"threads_started":1}
I20260925 15:23:08.494483 19229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:08.494252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.195:51471)
0925 15:23:08.494405 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:08.494437 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":95,"threads_started":1}
I20260925 15:23:08.494628 19230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:08.494384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.193:45579)
0925 15:23:08.494499 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:08.494575 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":72,"threads_started":1}
I20260925 15:23:08.687136 19231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:08.686917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.194:36395)
0925 15:23:08.687044 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:08.687090 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":79,"threads_started":1}
I20260925 15:23:09.495357 19232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:09.495029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.195:48823)
0925 15:23:09.495265 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:09.495308 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":169,"threads_started":1}
I20260925 15:23:09.495622 19233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:09.495284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.193:42041)
0925 15:23:09.495522 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:09.495570 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":73,"threads_started":1}
I20260925 15:23:09.687994 19234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:09.687721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.194:51979)
0925 15:23:09.687900 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:09.687945 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":96,"threads_started":1}
I20260925 15:23:10.496376 19235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:10.496105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.195:59927)
0925 15:23:10.496240 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:10.496284 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":79,"threads_started":1}
I20260925 15:23:10.496786 19236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:10.496451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.193:53017)
0925 15:23:10.496708 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:10.496748 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":91,"threads_started":1}
I20260925 15:23:10.688946 19237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:10.688729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.194:33385)
0925 15:23:10.688856 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:10.688900 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":62,"threads_started":1}
I20260925 15:23:11.497144 19238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:11.496897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.195:56275)
0925 15:23:11.497058 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:11.497097 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":65,"threads_started":1}
I20260925 15:23:11.497381 19239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:11.497193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.193:48699)
0925 15:23:11.497297 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:11.497333 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":66,"threads_started":1}
I20260925 15:23:11.689656 19240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:11.689445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.194:51325)
0925 15:23:11.689573 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:11.689607 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":85,"threads_started":1}
I20260925 15:23:12.498142 19241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:12.497838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.195:46111)
0925 15:23:12.498020 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:12.498086 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":80,"threads_started":1}
I20260925 15:23:12.498199 19242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:12.497963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.193:52379)
0925 15:23:12.498117 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:12.498158 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":68,"threads_started":1}
I20260925 15:23:12.690593 19243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:12.690292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.194:48075)
0925 15:23:12.690475 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:12.690525 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":78,"threads_started":1}
I20260925 15:23:13.498930 19244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:13.498729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.195:58901)
0925 15:23:13.498855 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:13.498886 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":57,"threads_started":1}
I20260925 15:23:13.498971 19245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:13.498784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.193:40827)
0925 15:23:13.498901 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:13.498930 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":54,"threads_started":1}
I20260925 15:23:13.691455 19246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:13.691152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.194:32833)
0925 15:23:13.691366 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:13.691410 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":91,"threads_started":1}
I20260925 15:23:14.499817 19247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:14.499578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.193:36327)
0925 15:23:14.499717 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:14.499760 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
I20260925 15:23:14.499847 19248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:14.499602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.195:57613)
0925 15:23:14.499756 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:14.499803 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":105,"threads_started":1}
I20260925 15:23:14.692507 19249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:14.692146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:34373 (local address 127.3.211.194:46955)
0925 15:23:14.692403 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:14.692459 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:34373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":68,"threads_started":1}
I20260925 15:23:15.036726 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 18756
I20260925 15:23:15.041803 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 18920
I20260925 15:23:15.046752 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 19055
2026-09-25T15:23:15Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18430 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:23:15 dist-test-slave-4bk3 krb5kdc[19253](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:23:15 dist-test-slave-4bk3 krb5kdc[19253](info): set up 2 sockets
Sep 25 15:23:15 dist-test-slave-4bk3 krb5kdc[19253](info): commencing operation
krb5kdc: starting...
W20260925 15:23:16.112583 3919 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Sep 25 15:23:16 dist-test-slave-4bk3 krb5kdc[19253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:23:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:23:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.211.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
I20260925 15:23:16.241223 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:45495
--webserver_interface=127.3.211.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:41461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.211.254:45495
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:23:16.324762 19266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:23:16.324993 19266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:23:16.325019 19266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:23:16.326360 19266 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 15:23:16.326462 19266 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 15:23:16.326485 19266 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 15:23:16.326519 19266 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:23:16.326539 19266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:23:16.326560 19266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:23:16.326743 19266 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:23:16.326771 19266 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:23:16.329161 19266 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:41461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.211.254:45495
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:45495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.211.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:23:16.329509 19266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:23:16.329792 19266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:23:16.335458 19271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:16.335466 19274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:16.335470 19272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:23:16.335489 19266 server_base.cc:1034] running on GCE node
I20260925 15:23:16.335899 19266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:23:16.336246 19266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:23:16.337428 19266 hybrid_clock.cc:648] HybridClock initialized: now 1790349796337406 us; error 35 us; skew 500 ppm
Sep 25 15:23:16 dist-test-slave-4bk3 krb5kdc[19253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349796, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:23:16.341125 19266 init.cc:377] Logged in from keytab as kudu/127.3.211.254@KRBTEST.COM (short username kudu)
I20260925 15:23:16.341763 19266 webserver.cc:460] Webserver started at http://127.3.211.254:37939/ using document root <none> and password file <none>
I20260925 15:23:16.342032 19266 fs_manager.cc:359] Metadata directory not provided
I20260925 15:23:16.342108 19266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:23:16.342279 19266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:23:16.343935 19266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/master-0/data/instance:
uuid: "486143fcc87440ea915817f8a7a43d07"
format_stamp: "Formatted at 2026-09-25 15:23:16 on dist-test-slave-4bk3"
server_key: "60b243b70c6f0aa2f81924954faba119"
server_key_iv: "b603dc795109e19ca615f908a76b7437"
server_key_version: "encryptionkey@0"
I20260925 15:23:16.344610 19266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/master-0/wal/instance:
uuid: "486143fcc87440ea915817f8a7a43d07"
format_stamp: "Formatted at 2026-09-25 15:23:16 on dist-test-slave-4bk3"
server_key: "60b243b70c6f0aa2f81924954faba119"
server_key_iv: "b603dc795109e19ca615f908a76b7437"
server_key_version: "encryptionkey@0"
I20260925 15:23:16.346613 19266 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:23:16.347867 19281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:16.348414 19266 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:23:16.348577 19266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/master-0/wal
uuid: "486143fcc87440ea915817f8a7a43d07"
format_stamp: "Formatted at 2026-09-25 15:23:16 on dist-test-slave-4bk3"
server_key: "60b243b70c6f0aa2f81924954faba119"
server_key_iv: "b603dc795109e19ca615f908a76b7437"
server_key_version: "encryptionkey@0"
I20260925 15:23:16.348696 19266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:23:16.373489 19266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:23:16.376089 19266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:23:16.376261 19266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:23:16.380615 19266 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.254:45495
I20260925 15:23:16.380697 19333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.254:45495 every 8 connection(s)
I20260925 15:23:16.381068 19266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
I20260925 15:23:16.381940 19334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:23:16.384187 19334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07: Bootstrap starting.
I20260925 15:23:16.384876 19334 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07: Neither blocks nor log segments found. Creating new log.
I20260925 15:23:16.385123 19334 log.cc:824] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07: Log is configured to *not* fsync() on all Append() calls
I20260925 15:23:16.385778 19334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07: No bootstrap required, opened a new log
I20260925 15:23:16.387097 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 19266
I20260925 15:23:16.387240 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/master-0/wal/instance
I20260925 15:23:16.387455 3919 external_mini_cluster.cc:1428] Setting key 4a98699d26452088d2330ebf65818b33
I20260925 15:23:16.387470 19334 raft_consensus.cc:348] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486143fcc87440ea915817f8a7a43d07" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 45495 } }
I20260925 15:23:16.387671 19334 raft_consensus.cc:374] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:23:16.387720 19334 raft_consensus.cc:729] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 486143fcc87440ea915817f8a7a43d07, State: Initialized, Role: FOLLOWER
I20260925 15:23:16.387872 19334 consensus_queue.cc:260] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486143fcc87440ea915817f8a7a43d07" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 45495 } }
I20260925 15:23:16.387975 19334 raft_consensus.cc:388] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:23:16.388008 19334 raft_consensus.cc:482] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:23:16.388053 19334 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:23:16.389002 19334 raft_consensus.cc:504] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486143fcc87440ea915817f8a7a43d07" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 45495 } }
I20260925 15:23:16.389132 19334 leader_election.cc:302] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 486143fcc87440ea915817f8a7a43d07; no voters:
I20260925 15:23:16.389338 19334 leader_election.cc:288] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:23:16.389369 19339 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:23:16.389531 19339 raft_consensus.cc:686] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07 [term 1 LEADER]: Becoming Leader. State: Replica: 486143fcc87440ea915817f8a7a43d07, State: Running, Role: LEADER
I20260925 15:23:16.389627 19334 sys_catalog.cc:564] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:23:16.389734 19339 consensus_queue.cc:237] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486143fcc87440ea915817f8a7a43d07" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 45495 } }
I20260925 15:23:16.390142 19339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 486143fcc87440ea915817f8a7a43d07. Latest consensus state: current_term: 1 leader_uuid: "486143fcc87440ea915817f8a7a43d07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486143fcc87440ea915817f8a7a43d07" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 45495 } } }
I20260925 15:23:16.390161 19341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "486143fcc87440ea915817f8a7a43d07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486143fcc87440ea915817f8a7a43d07" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 45495 } } }
I20260925 15:23:16.390280 19339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07 [sys.catalog]: This master's current role is: LEADER
I20260925 15:23:16.390480 19341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07 [sys.catalog]: This master's current role is: LEADER
I20260925 15:23:16.390628 19348 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:23:16.391127 19348 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:23:16.393330 19348 catalog_manager.cc:1347] Generated new cluster ID: 9e94e23d535d48fcb7ce57f993218fae
I20260925 15:23:16.393399 19348 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 15:23:16 dist-test-slave-4bk3 krb5kdc[19253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:23:16.400810 19338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:16.388047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52214 (local address 127.3.211.254:45495)
0925 15:23:16.388240 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:16.388250 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:16.388292 (+ 42us) server_negotiation.cc:407] Connection header received
0925 15:23:16.388454 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:16.388460 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:16.388515 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:16.388667 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:16.389389 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:16.390872 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:16.392073 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:16.392791 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:16.392918 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:16.395817 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:16.395822 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:16.395822 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:23:16.395834 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:16.398611 (+ 2777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:16.399066 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:16.399069 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:16.399070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:16.399187 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:16.399371 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:16.399371 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:16.399371 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:16.399713 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:16.400286 (+ 573us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:16.400398 (+ 112us) server_negotiation.cc:299] Negotiation successful
0925 15:23:16.400545 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":95,"threads_started":1}
I20260925 15:23:16.414533 19348 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:23:16.415479 19348 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:23:16.422514 19348 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07: Generated new TSK 0
I20260925 15:23:16.422749 19348 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.211.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab.
Entry for principal kudu/127.3.211.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab.
I20260925 15:23:16.452124 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.193:0
--local_ip_for_outbound_sockets=127.3.211.193
--webserver_interface=127.3.211.193
--webserver_port=0
--tserver_master_addrs=127.3.211.254:45495
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab
--principal=kudu/127.3.211.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:41461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:23:16.538497 19360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:23:16.538741 19360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:23:16.538767 19360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:23:16.540040 19360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:23:16.540099 19360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:23:16.540127 19360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.193
I20260925 15:23:16.541836 19360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:41461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab
--principal=kudu/127.3.211.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.211.193
--webserver_port=0
--tserver_master_addrs=127.3.211.254:45495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.193
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:23:16.542146 19360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:23:16.542428 19360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:23:16.547768 19368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:16.547784 19366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:23:16.548009 19360 server_base.cc:1034] running on GCE node
W20260925 15:23:16.547773 19365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:23:16.548305 19360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:23:16.548739 19360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:23:16.549908 19360 hybrid_clock.cc:648] HybridClock initialized: now 1790349796549896 us; error 31 us; skew 500 ppm
Sep 25 15:23:16 dist-test-slave-4bk3 krb5kdc[19253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349796, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:23:16.552954 19360 init.cc:377] Logged in from keytab as kudu/127.3.211.193@KRBTEST.COM (short username kudu)
I20260925 15:23:16.553439 19360 webserver.cc:460] Webserver started at http://127.3.211.193:36767/ using document root <none> and password file <none>
I20260925 15:23:16.553697 19360 fs_manager.cc:359] Metadata directory not provided
I20260925 15:23:16.553745 19360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:23:16.553882 19360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:23:16.554958 19360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-0/data/instance:
uuid: "e6429bbd8ad248bca7936b06a1925fc7"
format_stamp: "Formatted at 2026-09-25 15:23:16 on dist-test-slave-4bk3"
server_key: "66ffcc9194a142b7a57a5c419ec2e756"
server_key_iv: "2a0ad2d7fcf05d9e92cf94ad90ec9b0d"
server_key_version: "encryptionkey@0"
I20260925 15:23:16.555330 19360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-0/wal/instance:
uuid: "e6429bbd8ad248bca7936b06a1925fc7"
format_stamp: "Formatted at 2026-09-25 15:23:16 on dist-test-slave-4bk3"
server_key: "66ffcc9194a142b7a57a5c419ec2e756"
server_key_iv: "2a0ad2d7fcf05d9e92cf94ad90ec9b0d"
server_key_version: "encryptionkey@0"
I20260925 15:23:16.556816 19360 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:23:16.557715 19375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:16.557988 19360 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:23:16.558112 19360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-0/wal
uuid: "e6429bbd8ad248bca7936b06a1925fc7"
format_stamp: "Formatted at 2026-09-25 15:23:16 on dist-test-slave-4bk3"
server_key: "66ffcc9194a142b7a57a5c419ec2e756"
server_key_iv: "2a0ad2d7fcf05d9e92cf94ad90ec9b0d"
server_key_version: "encryptionkey@0"
I20260925 15:23:16.558194 19360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:23:16.576454 19360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:23:16.578840 19360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:23:16.579039 19360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:23:16.579298 19360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:23:16.579629 19360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:23:16.579685 19360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:16.579720 19360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:23:16.579751 19360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:16.585657 19360 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.193:44505
I20260925 15:23:16.585702 19488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.193:44505 every 8 connection(s)
I20260925 15:23:16.586095 19360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:23:16.587647 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 19360
I20260925 15:23:16.587781 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-0/wal/instance
I20260925 15:23:16.587980 3919 external_mini_cluster.cc:1428] Setting key 4cd5e6bbbe8b689d8f50766bb4e8cd7c
Sep 25 15:23:16 dist-test-slave-4bk3 krb5kdc[19253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349796, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.193@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.211.194@KRBTEST.COM; defaulting to no policy
I20260925 15:23:16.598078 19338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:16.587009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:57599 (local address 127.3.211.254:45495)
0925 15:23:16.587104 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:16.587107 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:16.587244 (+ 137us) server_negotiation.cc:407] Connection header received
0925 15:23:16.587812 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:16.587814 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:16.587834 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:16.587917 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:16.589306 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:16.590118 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:16.592043 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:16.592432 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:16.592542 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:16.594750 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:16.594752 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:16.594753 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:23:16.594763 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:16.596528 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:16.596957 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:16.596959 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:16.596959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:16.597023 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:16.597326 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:16.597327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:16.597328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:16.597489 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:16.597613 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:16.597934 (+ 321us) server_negotiation.cc:299] Negotiation successful
0925 15:23:16.598005 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 15:23:16.598309 19491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:16.586914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.193:57599)
0925 15:23:16.587100 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:16.587140 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:16.587657 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:16.587987 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:16.587991 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:16.588912 (+ 921us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:23:16.589192 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:16.589198 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:16.590430 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:16.590431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:16.591935 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:16.591938 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:16.592522 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:16.592523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:16.592655 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:16.593241 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:23:16.593250 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:23:16.594648 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:23:16.596632 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:23:16.596639 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:23:16.596641 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:23:16.596870 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:23:16.597119 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:23:16.597120 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:23:16.597120 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:23:16.597239 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:23:16.597639 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:23:16.597649 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:23:16.597795 (+ 146us) client_negotiation.cc:769] Sending connection context
0925 15:23:16.597897 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 15:23:16.598032 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":65,"threads_started":1}
I20260925 15:23:16.598953 19489 heartbeater.cc:347] Connected to a master server at 127.3.211.254:45495
I20260925 15:23:16.599042 19489 heartbeater.cc:464] Registering TS with master...
I20260925 15:23:16.599256 19489 heartbeater.cc:511] Master 127.3.211.254:45495 requested a full tablet report, sending...
I20260925 15:23:16.599795 19298 ts_manager.cc:194] Registered new tserver with Master: e6429bbd8ad248bca7936b06a1925fc7 (127.3.211.193:44505)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.194@KRBTEST.COM" created.
I20260925 15:23:16.600612 19298 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.193@KRBTEST.COM'} at 127.3.211.193:57599
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab.
Entry for principal kudu/127.3.211.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab.
I20260925 15:23:16.617286 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.194:0
--local_ip_for_outbound_sockets=127.3.211.194
--webserver_interface=127.3.211.194
--webserver_port=0
--tserver_master_addrs=127.3.211.254:45495
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab
--principal=kudu/127.3.211.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:41461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:23:16.705421 19494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:23:16.705719 19494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:23:16.705765 19494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:23:16.707020 19494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:23:16.707098 19494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:23:16.707146 19494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.194
I20260925 15:23:16.708778 19494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:41461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab
--principal=kudu/127.3.211.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.211.194
--webserver_port=0
--tserver_master_addrs=127.3.211.254:45495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.194
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:23:16.709084 19494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:23:16.709337 19494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:23:16.715195 19500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:16.715137 19499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:23:16.715124 19494 server_base.cc:1034] running on GCE node
W20260925 15:23:16.715157 19502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:23:16.716013 19494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:23:16.716271 19494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:23:16.717427 19494 hybrid_clock.cc:648] HybridClock initialized: now 1790349796717416 us; error 37 us; skew 500 ppm
Sep 25 15:23:16 dist-test-slave-4bk3 krb5kdc[19253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349796, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:23:16.728004 19494 init.cc:377] Logged in from keytab as kudu/127.3.211.194@KRBTEST.COM (short username kudu)
I20260925 15:23:16.728607 19494 webserver.cc:460] Webserver started at http://127.3.211.194:34113/ using document root <none> and password file <none>
I20260925 15:23:16.728894 19494 fs_manager.cc:359] Metadata directory not provided
I20260925 15:23:16.728978 19494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:23:16.729156 19494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:23:16.730327 19494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-1/data/instance:
uuid: "6746832d79eb409e948e2cd7f1742bfe"
format_stamp: "Formatted at 2026-09-25 15:23:16 on dist-test-slave-4bk3"
server_key: "2e109b40ef541a9710d704324e500792"
server_key_iv: "8e86b25b90e8c778dde8149850e2b800"
server_key_version: "encryptionkey@0"
I20260925 15:23:16.730739 19494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-1/wal/instance:
uuid: "6746832d79eb409e948e2cd7f1742bfe"
format_stamp: "Formatted at 2026-09-25 15:23:16 on dist-test-slave-4bk3"
server_key: "2e109b40ef541a9710d704324e500792"
server_key_iv: "8e86b25b90e8c778dde8149850e2b800"
server_key_version: "encryptionkey@0"
I20260925 15:23:16.732502 19494 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260925 15:23:16.733431 19509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:16.733651 19494 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:23:16.733764 19494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-1/wal
uuid: "6746832d79eb409e948e2cd7f1742bfe"
format_stamp: "Formatted at 2026-09-25 15:23:16 on dist-test-slave-4bk3"
server_key: "2e109b40ef541a9710d704324e500792"
server_key_iv: "8e86b25b90e8c778dde8149850e2b800"
server_key_version: "encryptionkey@0"
I20260925 15:23:16.733855 19494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:23:16.745239 19494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:23:16.747661 19494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:23:16.747835 19494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:23:16.748107 19494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:23:16.748466 19494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:23:16.748518 19494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:16.748562 19494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:23:16.748615 19494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:16.754944 19494 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.194:42381
I20260925 15:23:16.754998 19622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.194:42381 every 8 connection(s)
I20260925 15:23:16.755360 19494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
Sep 25 15:23:16 dist-test-slave-4bk3 krb5kdc[19253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349796, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.194@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:23:16.763887 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 19494
I20260925 15:23:16.764038 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-1/wal/instance
I20260925 15:23:16.764236 3919 external_mini_cluster.cc:1428] Setting key 043ab16ac57e30bd3afd2e18647a2db8
I20260925 15:23:16.766546 19338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:16.756207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:53835 (local address 127.3.211.254:45495)
0925 15:23:16.756276 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:16.756278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:16.756561 (+ 283us) server_negotiation.cc:407] Connection header received
0925 15:23:16.757051 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:16.757052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:16.757063 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:16.757152 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:16.758156 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:16.758675 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:16.759730 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:16.759928 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:16.760012 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:16.762455 (+ 2443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:16.762457 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:16.762457 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:23:16.762464 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:16.764239 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:16.765314 (+ 1075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:16.765316 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:16.765316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:16.765408 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:16.765772 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:16.765774 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:16.765775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:16.765915 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:16.766005 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:16.766334 (+ 329us) server_negotiation.cc:299] Negotiation successful
0925 15:23:16.766445 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 15:23:16.766791 19625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:16.756203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.194:53835)
0925 15:23:16.756422 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:16.756467 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:16.756947 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:16.757202 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:16.757205 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:16.757823 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:23:16.758028 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:16.758033 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:16.758786 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:16.758791 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:16.759601 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:16.759603 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:16.760046 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:16.760047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:16.760169 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:16.760850 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:23:16.760859 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:23:16.762351 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:23:16.764500 (+ 2149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:23:16.764517 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:23:16.764521 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:23:16.765185 (+ 664us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:23:16.765523 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:23:16.765524 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:23:16.765524 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:23:16.765670 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:23:16.765993 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:23:16.766000 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:23:16.766174 (+ 174us) client_negotiation.cc:769] Sending connection context
0925 15:23:16.766300 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 15:23:16.766494 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":89,"threads_started":1}
I20260925 15:23:16.767299 19623 heartbeater.cc:347] Connected to a master server at 127.3.211.254:45495
I20260925 15:23:16.767431 19623 heartbeater.cc:464] Registering TS with master...
I20260925 15:23:16.767645 19623 heartbeater.cc:511] Master 127.3.211.254:45495 requested a full tablet report, sending...
I20260925 15:23:16.768096 19298 ts_manager.cc:194] Registered new tserver with Master: 6746832d79eb409e948e2cd7f1742bfe (127.3.211.194:42381)
I20260925 15:23:16.768736 19298 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.194@KRBTEST.COM'} at 127.3.211.194:53835
WARNING: no policy specified for kudu/127.3.211.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab.
Entry for principal kudu/127.3.211.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab.
I20260925 15:23:16.792145 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.195:0
--local_ip_for_outbound_sockets=127.3.211.195
--webserver_interface=127.3.211.195
--webserver_port=0
--tserver_master_addrs=127.3.211.254:45495
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab
--principal=kudu/127.3.211.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:41461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:23:16.893826 19628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:23:16.894055 19628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:23:16.894091 19628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:23:16.895393 19628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:23:16.895450 19628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:23:16.895479 19628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.195
I20260925 15:23:16.897192 19628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:41461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab
--principal=kudu/127.3.211.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.211.195
--webserver_port=0
--tserver_master_addrs=127.3.211.254:45495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.195
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:23:16.897486 19628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:23:16.897704 19628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:23:16.902865 19636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:16.902876 19633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:23:16.902899 19628 server_base.cc:1034] running on GCE node
W20260925 15:23:16.902915 19634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:23:16.903337 19628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:23:16.903622 19628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:23:16.904769 19628 hybrid_clock.cc:648] HybridClock initialized: now 1790349796904750 us; error 31 us; skew 500 ppm
Sep 25 15:23:16 dist-test-slave-4bk3 krb5kdc[19253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349796, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:23:16.907898 19628 init.cc:377] Logged in from keytab as kudu/127.3.211.195@KRBTEST.COM (short username kudu)
I20260925 15:23:16.908552 19628 webserver.cc:460] Webserver started at http://127.3.211.195:39803/ using document root <none> and password file <none>
I20260925 15:23:16.908820 19628 fs_manager.cc:359] Metadata directory not provided
I20260925 15:23:16.908869 19628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:23:16.909008 19628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:23:16.910167 19628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-2/data/instance:
uuid: "f18797e136f44456af1360609527384d"
format_stamp: "Formatted at 2026-09-25 15:23:16 on dist-test-slave-4bk3"
server_key: "5989802ce37c4cff63c1cee4873e64c1"
server_key_iv: "68a0257ccbc4f193a585eb4e157036b1"
server_key_version: "encryptionkey@0"
I20260925 15:23:16.910557 19628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-2/wal/instance:
uuid: "f18797e136f44456af1360609527384d"
format_stamp: "Formatted at 2026-09-25 15:23:16 on dist-test-slave-4bk3"
server_key: "5989802ce37c4cff63c1cee4873e64c1"
server_key_iv: "68a0257ccbc4f193a585eb4e157036b1"
server_key_version: "encryptionkey@0"
I20260925 15:23:16.912247 19628 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:23:16.913317 19643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:16.913875 19628 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:23:16.913973 19628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-2/wal
uuid: "f18797e136f44456af1360609527384d"
format_stamp: "Formatted at 2026-09-25 15:23:16 on dist-test-slave-4bk3"
server_key: "5989802ce37c4cff63c1cee4873e64c1"
server_key_iv: "68a0257ccbc4f193a585eb4e157036b1"
server_key_version: "encryptionkey@0"
I20260925 15:23:16.914103 19628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:23:16.937254 19628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:23:16.939903 19628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:23:16.940084 19628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:23:16.940436 19628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:23:16.940796 19628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:23:16.940832 19628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:16.940856 19628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:23:16.940868 19628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:16.947503 19628 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.195:39309
I20260925 15:23:16.947553 19756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.195:39309 every 8 connection(s)
I20260925 15:23:16.947947 19628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
I20260925 15:23:16.948393 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 19628
I20260925 15:23:16.948505 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-2/wal/instance
I20260925 15:23:16.948700 3919 external_mini_cluster.cc:1428] Setting key 73a3aa06c95666d549ebe4cead144eeb
Sep 25 15:23:16 dist-test-slave-4bk3 krb5kdc[19253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349796, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.195@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:23:16.959293 19338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:16.948960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:42241 (local address 127.3.211.254:45495)
0925 15:23:16.949016 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:16.949018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:16.949307 (+ 289us) server_negotiation.cc:407] Connection header received
0925 15:23:16.949994 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:16.949995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:16.950007 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:16.950071 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:16.951120 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:16.951738 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:16.952991 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:16.953186 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:16.953256 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:16.955630 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:16.955632 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:16.955632 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:23:16.955640 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:16.957209 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:16.957673 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:16.957674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:16.957674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:16.957733 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:16.958129 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:16.958141 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:16.958141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:16.958409 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:16.958531 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:16.959111 (+ 580us) server_negotiation.cc:299] Negotiation successful
0925 15:23:16.959216 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 15:23:16.959435 19759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:16.948965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.195:42241)
0925 15:23:16.949148 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:16.949202 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:16.949848 (+ 646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:16.950134 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:16.950137 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:16.950760 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:23:16.950982 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:16.950987 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:16.951857 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:16.951858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:16.952892 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:16.952895 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:16.953271 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:16.953272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:16.953375 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:16.954005 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:23:16.954017 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:23:16.955500 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:23:16.957292 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:23:16.957300 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:23:16.957302 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:23:16.957557 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:23:16.957827 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:23:16.957828 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:23:16.957829 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:23:16.957946 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:23:16.958592 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:23:16.958606 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:23:16.958885 (+ 279us) client_negotiation.cc:769] Sending connection context
0925 15:23:16.959022 (+ 137us) client_negotiation.cc:240] Negotiation successful
0925 15:23:16.959184 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":97,"threads_started":1}
I20260925 15:23:16.960101 19757 heartbeater.cc:347] Connected to a master server at 127.3.211.254:45495
I20260925 15:23:16.960244 19757 heartbeater.cc:464] Registering TS with master...
I20260925 15:23:16.960505 19757 heartbeater.cc:511] Master 127.3.211.254:45495 requested a full tablet report, sending...
I20260925 15:23:16.961015 19298 ts_manager.cc:194] Registered new tserver with Master: f18797e136f44456af1360609527384d (127.3.211.195:39309)
I20260925 15:23:16.961421 19298 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.195@KRBTEST.COM'} at 127.3.211.195:42241
I20260925 15:23:16.971452 3919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:23:16.979102 19338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:16.972846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52222 (local address 127.3.211.254:45495)
0925 15:23:16.972912 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:16.972914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:16.973013 (+ 99us) server_negotiation.cc:407] Connection header received
0925 15:23:16.973070 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:16.973071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:16.973080 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:16.973134 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:16.973748 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:16.974231 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:16.974932 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:16.975111 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:16.975169 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:16.976471 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:16.976472 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:16.976473 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:23:16.976479 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:16.977920 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:16.978300 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:16.978301 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:16.978301 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:16.978358 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:16.978603 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:16.978604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:16.978604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:16.978768 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:16.978846 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:16.978965 (+ 119us) server_negotiation.cc:299] Negotiation successful
0925 15:23:16.979032 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 15:23:16.981395 19298 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52222:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:23:16.983000 19298 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:23:16.990998 19772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:16.986265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.195:39309 (local address 127.0.0.1:57206)
0925 15:23:16.986506 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:16.986518 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:16.986587 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:16.986928 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:16.986932 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:16.986934 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:23:16.987024 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:16.987028 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:16.988484 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:16.988485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:16.989635 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:16.989638 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:16.990573 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:16.990574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:16.990716 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:16.990735 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 15:23:16.990823 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 15:23:16.990890 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":68,"thread_start_us":77,"threads_started":1}
I20260925 15:23:16.991007 19773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:16.986494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57206 (local address 127.3.211.195:39309)
0925 15:23:16.986747 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:16.986751 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:16.986761 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:23:16.986802 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:16.986804 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:16.986819 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:16.987106 (+ 287us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:23:16.987199 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:16.988351 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:16.989731 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:16.990469 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:16.990562 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:16.990621 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:16.990848 (+ 227us) server_negotiation.cc:299] Negotiation successful
0925 15:23:16.990897 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":88,"threads_started":1}
I20260925 15:23:16.991330 19768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:16.986008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:44505 (local address 127.0.0.1:51808)
0925 15:23:16.986436 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:16.986651 (+ 215us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:16.986716 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:16.987007 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:16.987008 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:16.987009 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:23:16.987040 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:16.987041 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:16.988480 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:16.988482 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:16.990054 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:16.990058 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:16.990991 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:16.990992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:16.991068 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:16.991073 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:23:16.991200 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 15:23:16.991273 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"mutex_wait_us":150,"thread_start_us":95,"threads_started":1}
I20260925 15:23:16.991631 19770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:16.986155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51808 (local address 127.3.211.193:44505)
0925 15:23:16.986378 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:16.986384 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:16.986782 (+ 398us) server_negotiation.cc:407] Connection header received
0925 15:23:16.986827 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:16.986829 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:16.986847 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:16.986918 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:23:16.987121 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:16.988319 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:16.990171 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:16.990845 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:16.990887 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:16.990933 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:16.991497 (+ 564us) server_negotiation.cc:299] Negotiation successful
0925 15:23:16.991547 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":84,"threads_started":1}
I20260925 15:23:16.991858 19769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:16.986101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.194:42381 (local address 127.0.0.1:34826)
0925 15:23:16.986450 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:16.986496 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:16.986589 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:16.987158 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:16.987159 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:16.987162 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:23:16.987254 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:16.987257 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:16.988497 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:16.988497 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:16.990057 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:16.990059 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:16.991507 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:16.991510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:16.991569 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:16.991573 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:23:16.991652 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 15:23:16.991698 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":119,"thread_start_us":96,"threads_started":1}
I20260925 15:23:16.991873 19771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:16.986222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34826 (local address 127.3.211.194:42381)
0925 15:23:16.986456 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:16.986459 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:16.986572 (+ 113us) server_negotiation.cc:407] Connection header received
0925 15:23:16.986723 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:16.986726 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:16.986743 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:16.987001 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:23:16.987350 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:16.988285 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:16.990137 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:16.990912 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:16.990982 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:16.991044 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:16.991669 (+ 625us) server_negotiation.cc:299] Negotiation successful
0925 15:23:16.991792 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":105,"threads_started":1}
I20260925 15:23:16.991884 19691 tablet_service.cc:1467] Processing CreateTablet for tablet 93699c0f31cd45fc90c9b926ba1e7118 (DEFAULT_TABLE table=test-table [id=7e2935a6255a483fa99d85c561a832c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:23:16.992288 19691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93699c0f31cd45fc90c9b926ba1e7118. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:23:16.992194 19423 tablet_service.cc:1467] Processing CreateTablet for tablet 93699c0f31cd45fc90c9b926ba1e7118 (DEFAULT_TABLE table=test-table [id=7e2935a6255a483fa99d85c561a832c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:23:16.992765 19423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93699c0f31cd45fc90c9b926ba1e7118. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:23:16.992961 19557 tablet_service.cc:1467] Processing CreateTablet for tablet 93699c0f31cd45fc90c9b926ba1e7118 (DEFAULT_TABLE table=test-table [id=7e2935a6255a483fa99d85c561a832c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:23:16.993269 19557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93699c0f31cd45fc90c9b926ba1e7118. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:23:16.996066 19774 tablet_bootstrap.cc:492] T 93699c0f31cd45fc90c9b926ba1e7118 P 6746832d79eb409e948e2cd7f1742bfe: Bootstrap starting.
I20260925 15:23:16.996367 19776 tablet_bootstrap.cc:492] T 93699c0f31cd45fc90c9b926ba1e7118 P e6429bbd8ad248bca7936b06a1925fc7: Bootstrap starting.
I20260925 15:23:16.996881 19774 tablet_bootstrap.cc:654] T 93699c0f31cd45fc90c9b926ba1e7118 P 6746832d79eb409e948e2cd7f1742bfe: Neither blocks nor log segments found. Creating new log.
I20260925 15:23:16.996953 19776 tablet_bootstrap.cc:654] T 93699c0f31cd45fc90c9b926ba1e7118 P e6429bbd8ad248bca7936b06a1925fc7: Neither blocks nor log segments found. Creating new log.
I20260925 15:23:16.997416 19774 log.cc:824] T 93699c0f31cd45fc90c9b926ba1e7118 P 6746832d79eb409e948e2cd7f1742bfe: Log is configured to *not* fsync() on all Append() calls
I20260925 15:23:16.997469 19776 log.cc:824] T 93699c0f31cd45fc90c9b926ba1e7118 P e6429bbd8ad248bca7936b06a1925fc7: Log is configured to *not* fsync() on all Append() calls
I20260925 15:23:16.997882 19775 tablet_bootstrap.cc:492] T 93699c0f31cd45fc90c9b926ba1e7118 P f18797e136f44456af1360609527384d: Bootstrap starting.
I20260925 15:23:16.998456 19776 tablet_bootstrap.cc:492] T 93699c0f31cd45fc90c9b926ba1e7118 P e6429bbd8ad248bca7936b06a1925fc7: No bootstrap required, opened a new log
I20260925 15:23:16.998539 19776 ts_tablet_manager.cc:1397] T 93699c0f31cd45fc90c9b926ba1e7118 P e6429bbd8ad248bca7936b06a1925fc7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:23:16.998605 19775 tablet_bootstrap.cc:654] T 93699c0f31cd45fc90c9b926ba1e7118 P f18797e136f44456af1360609527384d: Neither blocks nor log segments found. Creating new log.
I20260925 15:23:16.998872 19774 tablet_bootstrap.cc:492] T 93699c0f31cd45fc90c9b926ba1e7118 P 6746832d79eb409e948e2cd7f1742bfe: No bootstrap required, opened a new log
I20260925 15:23:16.998955 19774 ts_tablet_manager.cc:1397] T 93699c0f31cd45fc90c9b926ba1e7118 P 6746832d79eb409e948e2cd7f1742bfe: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 15:23:16.999187 19775 log.cc:824] T 93699c0f31cd45fc90c9b926ba1e7118 P f18797e136f44456af1360609527384d: Log is configured to *not* fsync() on all Append() calls
I20260925 15:23:17.000015 19775 tablet_bootstrap.cc:492] T 93699c0f31cd45fc90c9b926ba1e7118 P f18797e136f44456af1360609527384d: No bootstrap required, opened a new log
I20260925 15:23:17.000103 19775 ts_tablet_manager.cc:1397] T 93699c0f31cd45fc90c9b926ba1e7118 P f18797e136f44456af1360609527384d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:23:17.000831 19774 raft_consensus.cc:348] T 93699c0f31cd45fc90c9b926ba1e7118 P 6746832d79eb409e948e2cd7f1742bfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6429bbd8ad248bca7936b06a1925fc7" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 44505 } } peers { permanent_uuid: "f18797e136f44456af1360609527384d" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 39309 } } peers { permanent_uuid: "6746832d79eb409e948e2cd7f1742bfe" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 42381 } }
I20260925 15:23:17.000828 19776 raft_consensus.cc:348] T 93699c0f31cd45fc90c9b926ba1e7118 P e6429bbd8ad248bca7936b06a1925fc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6429bbd8ad248bca7936b06a1925fc7" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 44505 } } peers { permanent_uuid: "f18797e136f44456af1360609527384d" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 39309 } } peers { permanent_uuid: "6746832d79eb409e948e2cd7f1742bfe" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 42381 } }
I20260925 15:23:17.000999 19776 raft_consensus.cc:374] T 93699c0f31cd45fc90c9b926ba1e7118 P e6429bbd8ad248bca7936b06a1925fc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:23:17.001010 19774 raft_consensus.cc:374] T 93699c0f31cd45fc90c9b926ba1e7118 P 6746832d79eb409e948e2cd7f1742bfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:23:17.001036 19776 raft_consensus.cc:729] T 93699c0f31cd45fc90c9b926ba1e7118 P e6429bbd8ad248bca7936b06a1925fc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6429bbd8ad248bca7936b06a1925fc7, State: Initialized, Role: FOLLOWER
I20260925 15:23:17.001056 19774 raft_consensus.cc:729] T 93699c0f31cd45fc90c9b926ba1e7118 P 6746832d79eb409e948e2cd7f1742bfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6746832d79eb409e948e2cd7f1742bfe, State: Initialized, Role: FOLLOWER
I20260925 15:23:17.001169 19774 consensus_queue.cc:260] T 93699c0f31cd45fc90c9b926ba1e7118 P 6746832d79eb409e948e2cd7f1742bfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6429bbd8ad248bca7936b06a1925fc7" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 44505 } } peers { permanent_uuid: "f18797e136f44456af1360609527384d" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 39309 } } peers { permanent_uuid: "6746832d79eb409e948e2cd7f1742bfe" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 42381 } }
I20260925 15:23:17.001241 19776 consensus_queue.cc:260] T 93699c0f31cd45fc90c9b926ba1e7118 P e6429bbd8ad248bca7936b06a1925fc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6429bbd8ad248bca7936b06a1925fc7" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 44505 } } peers { permanent_uuid: "f18797e136f44456af1360609527384d" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 39309 } } peers { permanent_uuid: "6746832d79eb409e948e2cd7f1742bfe" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 42381 } }
I20260925 15:23:17.001492 19774 ts_tablet_manager.cc:1428] T 93699c0f31cd45fc90c9b926ba1e7118 P 6746832d79eb409e948e2cd7f1742bfe: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 15:23:17.001627 19623 heartbeater.cc:503] Master 127.3.211.254:45495 was elected leader, sending a full tablet report...
I20260925 15:23:17.001628 19489 heartbeater.cc:503] Master 127.3.211.254:45495 was elected leader, sending a full tablet report...
I20260925 15:23:17.001541 19776 ts_tablet_manager.cc:1428] T 93699c0f31cd45fc90c9b926ba1e7118 P e6429bbd8ad248bca7936b06a1925fc7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:23:17.002251 19775 raft_consensus.cc:348] T 93699c0f31cd45fc90c9b926ba1e7118 P f18797e136f44456af1360609527384d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6429bbd8ad248bca7936b06a1925fc7" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 44505 } } peers { permanent_uuid: "f18797e136f44456af1360609527384d" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 39309 } } peers { permanent_uuid: "6746832d79eb409e948e2cd7f1742bfe" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 42381 } }
I20260925 15:23:17.002424 19775 raft_consensus.cc:374] T 93699c0f31cd45fc90c9b926ba1e7118 P f18797e136f44456af1360609527384d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:23:17.002470 19775 raft_consensus.cc:729] T 93699c0f31cd45fc90c9b926ba1e7118 P f18797e136f44456af1360609527384d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f18797e136f44456af1360609527384d, State: Initialized, Role: FOLLOWER
I20260925 15:23:17.002609 19775 consensus_queue.cc:260] T 93699c0f31cd45fc90c9b926ba1e7118 P f18797e136f44456af1360609527384d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6429bbd8ad248bca7936b06a1925fc7" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 44505 } } peers { permanent_uuid: "f18797e136f44456af1360609527384d" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 39309 } } peers { permanent_uuid: "6746832d79eb409e948e2cd7f1742bfe" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 42381 } }
I20260925 15:23:17.002905 19775 ts_tablet_manager.cc:1428] T 93699c0f31cd45fc90c9b926ba1e7118 P f18797e136f44456af1360609527384d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:23:17.002923 19757 heartbeater.cc:503] Master 127.3.211.254:45495 was elected leader, sending a full tablet report...
W20260925 15:23:17.005950 19624 tablet.cc:2369] T 93699c0f31cd45fc90c9b926ba1e7118 P 6746832d79eb409e948e2cd7f1742bfe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:23:17.053653 19781 raft_consensus.cc:482] T 93699c0f31cd45fc90c9b926ba1e7118 P e6429bbd8ad248bca7936b06a1925fc7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:23:17.053865 19781 raft_consensus.cc:504] T 93699c0f31cd45fc90c9b926ba1e7118 P e6429bbd8ad248bca7936b06a1925fc7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6429bbd8ad248bca7936b06a1925fc7" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 44505 } } peers { permanent_uuid: "f18797e136f44456af1360609527384d" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 39309 } } peers { permanent_uuid: "6746832d79eb409e948e2cd7f1742bfe" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 42381 } }
I20260925 15:23:17.054276 19781 leader_election.cc:288] T 93699c0f31cd45fc90c9b926ba1e7118 P e6429bbd8ad248bca7936b06a1925fc7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f18797e136f44456af1360609527384d (127.3.211.195:39309), 6746832d79eb409e948e2cd7f1742bfe (127.3.211.194:42381)
I20260925 15:23:17.058068 19491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:17.054520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.195:39309 (local address 127.3.211.193:56075)
0925 15:23:17.054601 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:17.054614 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:17.054659 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:17.054897 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:17.054898 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:17.054901 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:23:17.054984 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:17.054987 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:17.055698 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:17.055699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:17.056576 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:17.056579 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:17.057700 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:17.057701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:17.057806 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:17.057811 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:23:17.057888 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 15:23:17.057960 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":1}
I20260925 15:23:17.058068 19773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:17.054638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:56075 (local address 127.3.211.195:39309)
0925 15:23:17.054682 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:17.054685 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:17.054695 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:23:17.054727 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:17.054728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:17.054739 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:17.054830 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:23:17.055071 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:17.055609 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:17.056673 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:17.057479 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:17.057647 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:17.057663 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:17.057899 (+ 236us) server_negotiation.cc:299] Negotiation successful
0925 15:23:17.057960 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 15:23:17.058737 19783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:17.054525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.194:42381 (local address 127.3.211.193:50247)
0925 15:23:17.054767 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:17.054777 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:17.054842 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:17.055001 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:17.055002 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:17.055003 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:23:17.055107 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:17.055109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:17.055812 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:17.055813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:17.057478 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:17.057486 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:17.058360 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:17.058362 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:17.058454 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:17.058460 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 15:23:17.058541 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 15:23:17.058597 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":52,"thread_start_us":122,"threads_started":1}
I20260925 15:23:17.058732 19771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:17.054631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:50247 (local address 127.3.211.194:42381)
0925 15:23:17.054682 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:17.054684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:17.054873 (+ 189us) server_negotiation.cc:407] Connection header received
0925 15:23:17.054904 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:17.054905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:17.054915 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:17.054990 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:23:17.055183 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:17.055702 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:17.057692 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:17.058183 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:17.058291 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:17.058305 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:17.058578 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 15:23:17.058639 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 15:23:17.058787 19711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93699c0f31cd45fc90c9b926ba1e7118" candidate_uuid: "e6429bbd8ad248bca7936b06a1925fc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f18797e136f44456af1360609527384d" is_pre_election: true
I20260925 15:23:17.058971 19711 raft_consensus.cc:2454] T 93699c0f31cd45fc90c9b926ba1e7118 P f18797e136f44456af1360609527384d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6429bbd8ad248bca7936b06a1925fc7 in term 0.
I20260925 15:23:17.059048 19577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93699c0f31cd45fc90c9b926ba1e7118" candidate_uuid: "e6429bbd8ad248bca7936b06a1925fc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6746832d79eb409e948e2cd7f1742bfe" is_pre_election: true
I20260925 15:23:17.059185 19577 raft_consensus.cc:2454] T 93699c0f31cd45fc90c9b926ba1e7118 P 6746832d79eb409e948e2cd7f1742bfe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6429bbd8ad248bca7936b06a1925fc7 in term 0.
I20260925 15:23:17.059267 19378 leader_election.cc:302] T 93699c0f31cd45fc90c9b926ba1e7118 P e6429bbd8ad248bca7936b06a1925fc7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6429bbd8ad248bca7936b06a1925fc7, f18797e136f44456af1360609527384d; no voters:
I20260925 15:23:17.059424 19781 raft_consensus.cc:2781] T 93699c0f31cd45fc90c9b926ba1e7118 P e6429bbd8ad248bca7936b06a1925fc7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:23:17.059512 19781 raft_consensus.cc:482] T 93699c0f31cd45fc90c9b926ba1e7118 P e6429bbd8ad248bca7936b06a1925fc7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:23:17.059545 19781 raft_consensus.cc:3037] T 93699c0f31cd45fc90c9b926ba1e7118 P e6429bbd8ad248bca7936b06a1925fc7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:23:17.060385 19781 raft_consensus.cc:504] T 93699c0f31cd45fc90c9b926ba1e7118 P e6429bbd8ad248bca7936b06a1925fc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6429bbd8ad248bca7936b06a1925fc7" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 44505 } } peers { permanent_uuid: "f18797e136f44456af1360609527384d" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 39309 } } peers { permanent_uuid: "6746832d79eb409e948e2cd7f1742bfe" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 42381 } }
I20260925 15:23:17.060643 19781 leader_election.cc:288] T 93699c0f31cd45fc90c9b926ba1e7118 P e6429bbd8ad248bca7936b06a1925fc7 [CANDIDATE]: Term 1 election: Requested vote from peers f18797e136f44456af1360609527384d (127.3.211.195:39309), 6746832d79eb409e948e2cd7f1742bfe (127.3.211.194:42381)
I20260925 15:23:17.060734 19577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93699c0f31cd45fc90c9b926ba1e7118" candidate_uuid: "e6429bbd8ad248bca7936b06a1925fc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6746832d79eb409e948e2cd7f1742bfe"
I20260925 15:23:17.060807 19577 raft_consensus.cc:3037] T 93699c0f31cd45fc90c9b926ba1e7118 P 6746832d79eb409e948e2cd7f1742bfe [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:23:17.060832 19711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93699c0f31cd45fc90c9b926ba1e7118" candidate_uuid: "e6429bbd8ad248bca7936b06a1925fc7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f18797e136f44456af1360609527384d"
I20260925 15:23:17.060904 19711 raft_consensus.cc:3037] T 93699c0f31cd45fc90c9b926ba1e7118 P f18797e136f44456af1360609527384d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:23:17.061800 19577 raft_consensus.cc:2454] T 93699c0f31cd45fc90c9b926ba1e7118 P 6746832d79eb409e948e2cd7f1742bfe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6429bbd8ad248bca7936b06a1925fc7 in term 1.
I20260925 15:23:17.061806 19711 raft_consensus.cc:2454] T 93699c0f31cd45fc90c9b926ba1e7118 P f18797e136f44456af1360609527384d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6429bbd8ad248bca7936b06a1925fc7 in term 1.
I20260925 15:23:17.062083 19376 leader_election.cc:302] T 93699c0f31cd45fc90c9b926ba1e7118 P e6429bbd8ad248bca7936b06a1925fc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6746832d79eb409e948e2cd7f1742bfe, e6429bbd8ad248bca7936b06a1925fc7; no voters:
I20260925 15:23:17.062206 19781 raft_consensus.cc:2781] T 93699c0f31cd45fc90c9b926ba1e7118 P e6429bbd8ad248bca7936b06a1925fc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:23:17.062327 19781 raft_consensus.cc:686] T 93699c0f31cd45fc90c9b926ba1e7118 P e6429bbd8ad248bca7936b06a1925fc7 [term 1 LEADER]: Becoming Leader. State: Replica: e6429bbd8ad248bca7936b06a1925fc7, State: Running, Role: LEADER
I20260925 15:23:17.062427 19781 consensus_queue.cc:237] T 93699c0f31cd45fc90c9b926ba1e7118 P e6429bbd8ad248bca7936b06a1925fc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6429bbd8ad248bca7936b06a1925fc7" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 44505 } } peers { permanent_uuid: "f18797e136f44456af1360609527384d" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 39309 } } peers { permanent_uuid: "6746832d79eb409e948e2cd7f1742bfe" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 42381 } }
I20260925 15:23:17.063304 19298 catalog_manager.cc:5495] T 93699c0f31cd45fc90c9b926ba1e7118 P e6429bbd8ad248bca7936b06a1925fc7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6429bbd8ad248bca7936b06a1925fc7 (127.3.211.193). New cstate: current_term: 1 leader_uuid: "e6429bbd8ad248bca7936b06a1925fc7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6429bbd8ad248bca7936b06a1925fc7" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 44505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f18797e136f44456af1360609527384d" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 39309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6746832d79eb409e948e2cd7f1742bfe" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 42381 } health_report { overall_health: UNKNOWN } } }
I20260925 15:23:17.085987 19770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:17.083130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51814 (local address 127.3.211.193:44505)
0925 15:23:17.083181 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:17.083183 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:17.083192 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:23:17.083285 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:17.083287 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:17.083307 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:17.083383 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:23:17.083548 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:17.084070 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:17.084994 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:17.085198 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:17.085265 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:17.085496 (+ 231us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:23:17.085631 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:23:17.085710 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:17.085811 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 15:23:17.085832 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
W20260925 15:23:17.086841 19490 tablet.cc:2369] T 93699c0f31cd45fc90c9b926ba1e7118 P e6429bbd8ad248bca7936b06a1925fc7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:23:17.087963 19711 raft_consensus.cc:1261] T 93699c0f31cd45fc90c9b926ba1e7118 P f18797e136f44456af1360609527384d [term 1 FOLLOWER]: Refusing update from remote peer e6429bbd8ad248bca7936b06a1925fc7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:23:17.087968 19577 raft_consensus.cc:1261] T 93699c0f31cd45fc90c9b926ba1e7118 P 6746832d79eb409e948e2cd7f1742bfe [term 1 FOLLOWER]: Refusing update from remote peer e6429bbd8ad248bca7936b06a1925fc7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:23:17.088400 19784 consensus_queue.cc:1035] T 93699c0f31cd45fc90c9b926ba1e7118 P e6429bbd8ad248bca7936b06a1925fc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f18797e136f44456af1360609527384d" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:23:17.088583 19781 consensus_queue.cc:1035] T 93699c0f31cd45fc90c9b926ba1e7118 P e6429bbd8ad248bca7936b06a1925fc7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6746832d79eb409e948e2cd7f1742bfe" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 42381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:23:17.091632 19789 mvcc.cc:205] Tried to move back new op lower bound from 7333272768869457920 to 7333272768768356352. Current Snapshot: MvccSnapshot[applied={T|T < 7333272768869457920}]
I20260925 15:23:17.093461 19297 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:23:17.093616 19297 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:23:17.094434 19297 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07: Sending DeleteTablet for 3 replicas of tablet 93699c0f31cd45fc90c9b926ba1e7118
I20260925 15:23:17.094756 19423 tablet_service.cc:1514] Processing DeleteTablet for tablet 93699c0f31cd45fc90c9b926ba1e7118 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:23:17 UTC) from {username='kudu', principal='kudu/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:51808
I20260925 15:23:17.094811 19691 tablet_service.cc:1514] Processing DeleteTablet for tablet 93699c0f31cd45fc90c9b926ba1e7118 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:23:17 UTC) from {username='kudu', principal='kudu/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:57206
I20260925 15:23:17.094813 19557 tablet_service.cc:1514] Processing DeleteTablet for tablet 93699c0f31cd45fc90c9b926ba1e7118 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:23:17 UTC) from {username='kudu', principal='kudu/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:34826
I20260925 15:23:17.095018 19798 tablet_replica.cc:330] T 93699c0f31cd45fc90c9b926ba1e7118 P f18797e136f44456af1360609527384d: stopping tablet replica
I20260925 15:23:17.095125 19798 raft_consensus.cc:2229] T 93699c0f31cd45fc90c9b926ba1e7118 P f18797e136f44456af1360609527384d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:23:17.095247 19798 raft_consensus.cc:2258] T 93699c0f31cd45fc90c9b926ba1e7118 P f18797e136f44456af1360609527384d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:23:17.095240 19799 tablet_replica.cc:330] T 93699c0f31cd45fc90c9b926ba1e7118 P 6746832d79eb409e948e2cd7f1742bfe: stopping tablet replica
I20260925 15:23:17.095301 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 19360
I20260925 15:23:17.095954 19799 raft_consensus.cc:2229] T 93699c0f31cd45fc90c9b926ba1e7118 P 6746832d79eb409e948e2cd7f1742bfe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:23:17.096181 19799 raft_consensus.cc:2258] T 93699c0f31cd45fc90c9b926ba1e7118 P 6746832d79eb409e948e2cd7f1742bfe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:23:17.096596 19798 ts_tablet_manager.cc:1905] T 93699c0f31cd45fc90c9b926ba1e7118 P f18797e136f44456af1360609527384d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:23:17.097960 19799 ts_tablet_manager.cc:1905] T 93699c0f31cd45fc90c9b926ba1e7118 P 6746832d79eb409e948e2cd7f1742bfe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:23:17.098261 19798 ts_tablet_manager.cc:1918] T 93699c0f31cd45fc90c9b926ba1e7118 P f18797e136f44456af1360609527384d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:23:17.098356 19798 log.cc:1196] T 93699c0f31cd45fc90c9b926ba1e7118 P f18797e136f44456af1360609527384d: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-2/wal/wals/93699c0f31cd45fc90c9b926ba1e7118
I20260925 15:23:17.098642 19798 ts_tablet_manager.cc:1939] T 93699c0f31cd45fc90c9b926ba1e7118 P f18797e136f44456af1360609527384d: Deleting consensus metadata
I20260925 15:23:17.099169 19284 catalog_manager.cc:4841] TS f18797e136f44456af1360609527384d (127.3.211.195:39309): tablet 93699c0f31cd45fc90c9b926ba1e7118 (table test-table [id=7e2935a6255a483fa99d85c561a832c0]) successfully deleted
I20260925 15:23:17.099632 19799 ts_tablet_manager.cc:1918] T 93699c0f31cd45fc90c9b926ba1e7118 P 6746832d79eb409e948e2cd7f1742bfe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:23:17.099740 19799 log.cc:1196] T 93699c0f31cd45fc90c9b926ba1e7118 P 6746832d79eb409e948e2cd7f1742bfe: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-1/wal/wals/93699c0f31cd45fc90c9b926ba1e7118
I20260925 15:23:17.100026 19799 ts_tablet_manager.cc:1939] T 93699c0f31cd45fc90c9b926ba1e7118 P 6746832d79eb409e948e2cd7f1742bfe: Deleting consensus metadata
I20260925 15:23:17.100508 19282 catalog_manager.cc:4841] TS 6746832d79eb409e948e2cd7f1742bfe (127.3.211.194:42381): tablet 93699c0f31cd45fc90c9b926ba1e7118 (table test-table [id=7e2935a6255a483fa99d85c561a832c0]) successfully deleted
W20260925 15:23:17.101375 19284 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.211.193:44505 (error 108)
I20260925 15:23:17.101470 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 19494
I20260925 15:23:17.102247 19768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:17.102088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:44505 (local address 127.0.0.1:51830)
0925 15:23:17.102159 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:17.102204 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.193:44505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 15:23:17.102360 19284 catalog_manager.cc:4584] TS e6429bbd8ad248bca7936b06a1925fc7 (127.3.211.193:44505): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 93699c0f31cd45fc90c9b926ba1e7118: Network error: Client connection negotiation failed: client connection to 127.3.211.193:44505: connect: Connection refused (error 111)
I20260925 15:23:17.107470 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 19628
I20260925 15:23:17.112469 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 19266
I20260925 15:23:37.118422 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:45495
--webserver_interface=127.3.211.254
--webserver_port=37939
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:41461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.211.254:45495
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:23:37.208272 19801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:23:37.208602 19801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:23:37.208648 19801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:23:37.209949 19801 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 15:23:37.210026 19801 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 15:23:37.210047 19801 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 15:23:37.210069 19801 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:23:37.210098 19801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:23:37.210129 19801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:23:37.210155 19801 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:23:37.210186 19801 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:23:37.211839 19801 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:41461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.211.254:45495
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:45495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.211.254
--webserver_port=37939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:23:37.212177 19801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:23:37.212471 19801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:23:37.222682 19807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:37.222684 19808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:23:37.222730 19801 server_base.cc:1034] running on GCE node
W20260925 15:23:37.222725 19810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:23:37.223245 19801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:23:37.223974 19801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:23:37.225190 19801 hybrid_clock.cc:648] HybridClock initialized: now 1790349817225179 us; error 64 us; skew 500 ppm
Sep 25 15:23:37 dist-test-slave-4bk3 krb5kdc[19253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349817, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:23:37.228647 19801 init.cc:377] Logged in from keytab as kudu/127.3.211.254@KRBTEST.COM (short username kudu)
I20260925 15:23:37.229385 19801 webserver.cc:460] Webserver started at http://127.3.211.254:37939/ using document root <none> and password file <none>
I20260925 15:23:37.229640 19801 fs_manager.cc:359] Metadata directory not provided
I20260925 15:23:37.229708 19801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:23:37.231559 19801 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:23:37.232529 19817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:37.232841 19801 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260925 15:23:37.232967 19801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/master-0/wal
uuid: "486143fcc87440ea915817f8a7a43d07"
format_stamp: "Formatted at 2026-09-25 15:23:16 on dist-test-slave-4bk3"
server_key: "60b243b70c6f0aa2f81924954faba119"
server_key_iv: "b603dc795109e19ca615f908a76b7437"
server_key_version: "encryptionkey@0"
I20260925 15:23:37.233419 19801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:23:37.256964 19801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:23:37.259462 19801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:23:37.259585 19801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:23:37.263773 19801 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.254:45495
I20260925 15:23:37.263855 19869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.254:45495 every 8 connection(s)
I20260925 15:23:37.264323 19801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
I20260925 15:23:37.264786 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 19801
I20260925 15:23:37.265156 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.193:44505
--local_ip_for_outbound_sockets=127.3.211.193
--tserver_master_addrs=127.3.211.254:45495
--webserver_port=36767
--webserver_interface=127.3.211.193
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab
--principal=kudu/127.3.211.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:41461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:23:37.266768 19870 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:23:37.267535 19870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07: Bootstrap starting.
I20260925 15:23:37.276950 19870 log.cc:824] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07: Log is configured to *not* fsync() on all Append() calls
I20260925 15:23:37.279161 19870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:23:37.279518 19870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07: Bootstrap complete.
I20260925 15:23:37.282038 19870 raft_consensus.cc:348] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486143fcc87440ea915817f8a7a43d07" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 45495 } }
I20260925 15:23:37.282334 19870 raft_consensus.cc:729] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 486143fcc87440ea915817f8a7a43d07, State: Initialized, Role: FOLLOWER
I20260925 15:23:37.282552 19870 consensus_queue.cc:260] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486143fcc87440ea915817f8a7a43d07" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 45495 } }
I20260925 15:23:37.282655 19870 raft_consensus.cc:388] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:23:37.282717 19870 raft_consensus.cc:482] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:23:37.282768 19870 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07 [term 1 FOLLOWER]: Advancing to term 2
I20260925 15:23:37.283897 19870 raft_consensus.cc:504] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486143fcc87440ea915817f8a7a43d07" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 45495 } }
I20260925 15:23:37.284037 19870 leader_election.cc:302] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 486143fcc87440ea915817f8a7a43d07; no voters:
I20260925 15:23:37.284274 19870 leader_election.cc:288] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 15:23:37.284312 19874 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 15:23:37.284482 19874 raft_consensus.cc:686] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07 [term 2 LEADER]: Becoming Leader. State: Replica: 486143fcc87440ea915817f8a7a43d07, State: Running, Role: LEADER
I20260925 15:23:37.284590 19870 sys_catalog.cc:564] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:23:37.284575 19874 consensus_queue.cc:237] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486143fcc87440ea915817f8a7a43d07" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 45495 } }
I20260925 15:23:37.284912 19874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 486143fcc87440ea915817f8a7a43d07. Latest consensus state: current_term: 2 leader_uuid: "486143fcc87440ea915817f8a7a43d07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486143fcc87440ea915817f8a7a43d07" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 45495 } } }
I20260925 15:23:37.284987 19874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07 [sys.catalog]: This master's current role is: LEADER
I20260925 15:23:37.285164 19875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "486143fcc87440ea915817f8a7a43d07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486143fcc87440ea915817f8a7a43d07" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 45495 } } }
I20260925 15:23:37.285332 19875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07 [sys.catalog]: This master's current role is: LEADER
I20260925 15:23:37.285246 19879 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:23:37.286293 19879 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:23:37.286921 19879 catalog_manager.cc:1259] Loaded cluster ID: 9e94e23d535d48fcb7ce57f993218fae
I20260925 15:23:37.286978 19879 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:23:37.288206 19879 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:23:37.288453 19879 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07: Loaded TSK: 0
I20260925 15:23:37.300554 19879 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07: Generated new TSK 1
I20260925 15:23:37.300698 19879 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 15:23:37.364622 19872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:23:37.364845 19872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:23:37.364872 19872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:23:37.366216 19872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:23:37.366272 19872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:23:37.366302 19872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.193
I20260925 15:23:37.367908 19872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:41461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab
--principal=kudu/127.3.211.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.193:44505
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.211.193
--webserver_port=36767
--tserver_master_addrs=127.3.211.254:45495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.193
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:23:37.368196 19872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:23:37.368498 19872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:23:37.375108 19900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:37.375108 19897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:37.375108 19898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:23:37.375660 19872 server_base.cc:1034] running on GCE node
I20260925 15:23:37.375916 19872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:23:37.376220 19872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:23:37.377408 19872 hybrid_clock.cc:648] HybridClock initialized: now 1790349817377388 us; error 34 us; skew 500 ppm
Sep 25 15:23:37 dist-test-slave-4bk3 krb5kdc[19253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349817, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:23:37.380699 19872 init.cc:377] Logged in from keytab as kudu/127.3.211.193@KRBTEST.COM (short username kudu)
I20260925 15:23:37.381218 19872 webserver.cc:460] Webserver started at http://127.3.211.193:36767/ using document root <none> and password file <none>
I20260925 15:23:37.381457 19872 fs_manager.cc:359] Metadata directory not provided
I20260925 15:23:37.381546 19872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:23:37.383193 19872 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:23:37.384078 19907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:37.384305 19872 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:23:37.384443 19872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-0/wal
uuid: "e6429bbd8ad248bca7936b06a1925fc7"
format_stamp: "Formatted at 2026-09-25 15:23:16 on dist-test-slave-4bk3"
server_key: "66ffcc9194a142b7a57a5c419ec2e756"
server_key_iv: "2a0ad2d7fcf05d9e92cf94ad90ec9b0d"
server_key_version: "encryptionkey@0"
I20260925 15:23:37.384897 19872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:23:37.401124 19872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:23:37.403623 19872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:23:37.403807 19872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:23:37.404066 19872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:23:37.404608 19914 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 15:23:37.406770 19872 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 15:23:37.406889 19872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 15:23:37.406932 19872 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 15:23:37.407655 19872 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 15:23:37.407720 19872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 15:23:37.407817 19914 tablet_bootstrap.cc:492] T 93699c0f31cd45fc90c9b926ba1e7118 P e6429bbd8ad248bca7936b06a1925fc7: Bootstrap starting.
I20260925 15:23:37.415843 19872 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.193:44505
I20260925 15:23:37.416301 19872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:23:37.419727 19914 log.cc:824] T 93699c0f31cd45fc90c9b926ba1e7118 P e6429bbd8ad248bca7936b06a1925fc7: Log is configured to *not* fsync() on all Append() calls
I20260925 15:23:37.420099 20021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.193:44505 every 8 connection(s)
I20260925 15:23:37.421530 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 19872
I20260925 15:23:37.421866 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.194:42381
--local_ip_for_outbound_sockets=127.3.211.194
--tserver_master_addrs=127.3.211.254:45495
--webserver_port=34113
--webserver_interface=127.3.211.194
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab
--principal=kudu/127.3.211.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:41461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:23:37.423125 19914 tablet_bootstrap.cc:492] T 93699c0f31cd45fc90c9b926ba1e7118 P e6429bbd8ad248bca7936b06a1925fc7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:23:37.424096 19914 tablet_bootstrap.cc:492] T 93699c0f31cd45fc90c9b926ba1e7118 P e6429bbd8ad248bca7936b06a1925fc7: Bootstrap complete.
I20260925 15:23:37.425400 19914 ts_tablet_manager.cc:1397] T 93699c0f31cd45fc90c9b926ba1e7118 P e6429bbd8ad248bca7936b06a1925fc7: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.013s
I20260925 15:23:37.426967 19914 raft_consensus.cc:348] T 93699c0f31cd45fc90c9b926ba1e7118 P e6429bbd8ad248bca7936b06a1925fc7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6429bbd8ad248bca7936b06a1925fc7" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 44505 } } peers { permanent_uuid: "f18797e136f44456af1360609527384d" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 39309 } } peers { permanent_uuid: "6746832d79eb409e948e2cd7f1742bfe" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 42381 } }
I20260925 15:23:37.427323 19914 raft_consensus.cc:729] T 93699c0f31cd45fc90c9b926ba1e7118 P e6429bbd8ad248bca7936b06a1925fc7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6429bbd8ad248bca7936b06a1925fc7, State: Initialized, Role: FOLLOWER
I20260925 15:23:37.427503 19914 consensus_queue.cc:260] T 93699c0f31cd45fc90c9b926ba1e7118 P e6429bbd8ad248bca7936b06a1925fc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6429bbd8ad248bca7936b06a1925fc7" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 44505 } } peers { permanent_uuid: "f18797e136f44456af1360609527384d" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 39309 } } peers { permanent_uuid: "6746832d79eb409e948e2cd7f1742bfe" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 42381 } }
I20260925 15:23:37.427798 19914 ts_tablet_manager.cc:1428] T 93699c0f31cd45fc90c9b926ba1e7118 P e6429bbd8ad248bca7936b06a1925fc7: Time spent starting tablet: real 0.002s user 0.001s sys 0.003s
Sep 25 15:23:37 dist-test-slave-4bk3 krb5kdc[19253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349817, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.193@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:23:37.436246 20025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:37.417473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:43095 (local address 127.3.211.254:45495)
0925 15:23:37.420109 (+ 2636us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:37.420118 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:37.420164 (+ 46us) server_negotiation.cc:407] Connection header received
0925 15:23:37.420383 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:37.420391 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:37.420470 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:37.420667 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:37.424207 (+ 3540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:37.425290 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:37.427075 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:37.428167 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:37.428312 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:37.431440 (+ 3128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:37.431446 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:37.431446 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:23:37.431460 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:37.433736 (+ 2276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:37.434367 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:37.434371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:37.434372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:37.434511 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:37.434840 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:37.434841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:37.434841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:37.435174 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:37.435252 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:37.435800 (+ 548us) server_negotiation.cc:299] Negotiation successful
0925 15:23:37.435967 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2578,"thread_start_us":98,"threads_started":1}
I20260925 15:23:37.436973 20024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:37.417404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.193:43095)
0925 15:23:37.417611 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:37.417841 (+ 230us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:37.418336 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:37.423217 (+ 4881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:37.423224 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:37.423649 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:23:37.423989 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:37.423998 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:37.425448 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:37.425449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:37.426938 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:37.426944 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:37.428303 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:37.428304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:37.428474 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:37.429185 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:23:37.429205 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:23:37.431292 (+ 2087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:23:37.433854 (+ 2562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:23:37.433865 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:23:37.433868 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:23:37.434253 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:23:37.434614 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:23:37.434615 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:23:37.434615 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:23:37.434759 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:23:37.435298 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:23:37.435305 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:23:37.435464 (+ 159us) client_negotiation.cc:769] Sending connection context
0925 15:23:37.435571 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 15:23:37.436738 (+ 1167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":120,"threads_started":1}
I20260925 15:23:37.438081 20022 heartbeater.cc:347] Connected to a master server at 127.3.211.254:45495
I20260925 15:23:37.438201 20022 heartbeater.cc:464] Registering TS with master...
I20260925 15:23:37.438406 20022 heartbeater.cc:511] Master 127.3.211.254:45495 requested a full tablet report, sending...
I20260925 15:23:37.439258 19834 ts_manager.cc:194] Registered new tserver with Master: e6429bbd8ad248bca7936b06a1925fc7 (127.3.211.193:44505)
I20260925 15:23:37.440639 19834 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.193@KRBTEST.COM'} at 127.3.211.193:43095
I20260925 15:23:37.441797 20022 heartbeater.cc:503] Master 127.3.211.254:45495 was elected leader, sending a full tablet report...
I20260925 15:23:37.448642 20030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:37.440210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33690 (local address 127.3.211.193:44505)
0925 15:23:37.444763 (+ 4553us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:37.444768 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:37.444780 (+ 12us) server_negotiation.cc:407] Connection header received
0925 15:23:37.444816 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:37.444817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:37.444832 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:37.444920 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:23:37.445187 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:37.445894 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:37.447614 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:37.448163 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:37.448246 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:37.448287 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:37.448534 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 15:23:37.448575 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4517,"thread_start_us":229,"threads_started":1}
I20260925 15:23:37.448666 20029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:37.439928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:44505 (local address 127.0.0.1:33690)
0925 15:23:37.440556 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:37.440600 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:37.440678 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:37.444962 (+ 4284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:37.444966 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:37.444969 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:23:37.445055 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:37.445057 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:37.446000 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:37.446001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:37.447443 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:37.447447 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:37.448275 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:37.448276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:37.448415 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:37.448430 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 15:23:37.448510 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 15:23:37.448577 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":100,"threads_started":1}
I20260925 15:23:37.449215 19953 tablet_service.cc:1514] Processing DeleteTablet for tablet 93699c0f31cd45fc90c9b926ba1e7118 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:23:17 UTC) from {username='kudu', principal='kudu/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:33690
I20260925 15:23:37.449884 20031 tablet_replica.cc:330] stopping tablet replica
I20260925 15:23:37.450016 20031 raft_consensus.cc:2229] T 93699c0f31cd45fc90c9b926ba1e7118 P e6429bbd8ad248bca7936b06a1925fc7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:23:37.450107 20031 raft_consensus.cc:2258] T 93699c0f31cd45fc90c9b926ba1e7118 P e6429bbd8ad248bca7936b06a1925fc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:23:37.450398 20031 ts_tablet_manager.cc:1905] T 93699c0f31cd45fc90c9b926ba1e7118 P e6429bbd8ad248bca7936b06a1925fc7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:23:37.452481 20031 ts_tablet_manager.cc:1918] T 93699c0f31cd45fc90c9b926ba1e7118 P e6429bbd8ad248bca7936b06a1925fc7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:23:37.452569 20031 log.cc:1196] T 93699c0f31cd45fc90c9b926ba1e7118 P e6429bbd8ad248bca7936b06a1925fc7: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-0/wal/wals/93699c0f31cd45fc90c9b926ba1e7118
I20260925 15:23:37.452801 20031 ts_tablet_manager.cc:1939] T 93699c0f31cd45fc90c9b926ba1e7118 P e6429bbd8ad248bca7936b06a1925fc7: Deleting consensus metadata
I20260925 15:23:37.453356 19820 catalog_manager.cc:4841] TS e6429bbd8ad248bca7936b06a1925fc7 (127.3.211.193:44505): tablet 93699c0f31cd45fc90c9b926ba1e7118 (table test-table [id=7e2935a6255a483fa99d85c561a832c0]) successfully deleted
W20260925 15:23:37.518518 20027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:23:37.518779 20027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:23:37.518808 20027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:23:37.520071 20027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:23:37.520123 20027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:23:37.520150 20027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.194
I20260925 15:23:37.521736 20027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:41461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab
--principal=kudu/127.3.211.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.194:42381
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.211.194
--webserver_port=34113
--tserver_master_addrs=127.3.211.254:45495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.194
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:23:37.521961 20027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:23:37.522162 20027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:23:37.527851 20039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:37.527825 20037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:37.527825 20036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:23:37.528394 20027 server_base.cc:1034] running on GCE node
I20260925 15:23:37.528641 20027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:23:37.528954 20027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:23:37.530160 20027 hybrid_clock.cc:648] HybridClock initialized: now 1790349817530136 us; error 49 us; skew 500 ppm
Sep 25 15:23:37 dist-test-slave-4bk3 krb5kdc[19253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349817, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:23:37.533226 20027 init.cc:377] Logged in from keytab as kudu/127.3.211.194@KRBTEST.COM (short username kudu)
I20260925 15:23:37.533794 20027 webserver.cc:460] Webserver started at http://127.3.211.194:34113/ using document root <none> and password file <none>
I20260925 15:23:37.534065 20027 fs_manager.cc:359] Metadata directory not provided
I20260925 15:23:37.534132 20027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:23:37.535821 20027 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 15:23:37.537037 20046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:37.537422 20027 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 15:23:37.537540 20027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-1/wal
uuid: "6746832d79eb409e948e2cd7f1742bfe"
format_stamp: "Formatted at 2026-09-25 15:23:16 on dist-test-slave-4bk3"
server_key: "2e109b40ef541a9710d704324e500792"
server_key_iv: "8e86b25b90e8c778dde8149850e2b800"
server_key_version: "encryptionkey@0"
I20260925 15:23:37.537995 20027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:23:37.557761 20027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:23:37.560479 20027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:23:37.560611 20027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:23:37.560883 20027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:23:37.561237 20027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:23:37.561282 20027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:37.561306 20027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:23:37.561318 20027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:37.568073 20027 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.194:42381
I20260925 15:23:37.568117 20159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.194:42381 every 8 connection(s)
I20260925 15:23:37.568557 20027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
I20260925 15:23:37.569559 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 20027
I20260925 15:23:37.569861 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.195:39309
--local_ip_for_outbound_sockets=127.3.211.195
--tserver_master_addrs=127.3.211.254:45495
--webserver_port=39803
--webserver_interface=127.3.211.195
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab
--principal=kudu/127.3.211.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:41461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:23:37 dist-test-slave-4bk3 krb5kdc[19253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349817, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.194@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:23:37.578977 20025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:37.569405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:60893 (local address 127.3.211.254:45495)
0925 15:23:37.569467 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:37.569469 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:37.569669 (+ 200us) server_negotiation.cc:407] Connection header received
0925 15:23:37.570354 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:37.570355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:37.570369 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:37.570436 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:37.571465 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:37.572089 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:37.573266 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:37.573495 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:37.573587 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:37.575817 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:37.575819 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:37.575820 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:23:37.575829 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:37.577447 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:37.577921 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:37.577923 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:37.577923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:37.577990 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:37.578264 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:37.578265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:37.578266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:37.578394 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:37.578474 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:37.578782 (+ 308us) server_negotiation.cc:299] Negotiation successful
0925 15:23:37.578882 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 15:23:37.579190 20162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:37.569324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.194:60893)
0925 15:23:37.569542 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:37.569580 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:37.570235 (+ 655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:37.570508 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:37.570512 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:37.571116 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:23:37.571347 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:37.571351 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:37.572198 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:37.572199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:37.573187 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:37.573191 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:37.573600 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:37.573601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:37.573744 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:37.574320 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:23:37.574329 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:23:37.575712 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:23:37.577572 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:23:37.577581 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:23:37.577584 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:23:37.577828 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:23:37.578079 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:23:37.578081 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:23:37.578081 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:23:37.578195 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:23:37.578464 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:23:37.578470 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:23:37.578626 (+ 156us) client_negotiation.cc:769] Sending connection context
0925 15:23:37.578758 (+ 132us) client_negotiation.cc:240] Negotiation successful
0925 15:23:37.578933 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":95,"threads_started":1}
I20260925 15:23:37.579910 20160 heartbeater.cc:347] Connected to a master server at 127.3.211.254:45495
I20260925 15:23:37.579999 20160 heartbeater.cc:464] Registering TS with master...
I20260925 15:23:37.580216 20160 heartbeater.cc:511] Master 127.3.211.254:45495 requested a full tablet report, sending...
I20260925 15:23:37.580677 19834 ts_manager.cc:194] Registered new tserver with Master: 6746832d79eb409e948e2cd7f1742bfe (127.3.211.194:42381)
I20260925 15:23:37.581199 19834 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.194@KRBTEST.COM'} at 127.3.211.194:60893
W20260925 15:23:37.657819 20163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:23:37.658056 20163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:23:37.658083 20163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:23:37.659260 20163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:23:37.659303 20163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:23:37.659332 20163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.195
I20260925 15:23:37.660846 20163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:41461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab
--principal=kudu/127.3.211.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.195:39309
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.211.195
--webserver_port=39803
--tserver_master_addrs=127.3.211.254:45495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.195
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:23:37.661075 20163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:23:37.661284 20163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 15:23:37.667809 20163 server_base.cc:1034] running on GCE node
W20260925 15:23:37.667832 20168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:37.667783 20171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:37.667770 20169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:23:37.668476 20163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:23:37.668834 20163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:23:37.670034 20163 hybrid_clock.cc:648] HybridClock initialized: now 1790349817670004 us; error 73 us; skew 500 ppm
Sep 25 15:23:37 dist-test-slave-4bk3 krb5kdc[19253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349817, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:23:37.673859 20163 init.cc:377] Logged in from keytab as kudu/127.3.211.195@KRBTEST.COM (short username kudu)
I20260925 15:23:37.674610 20163 webserver.cc:460] Webserver started at http://127.3.211.195:39803/ using document root <none> and password file <none>
I20260925 15:23:37.675037 20163 fs_manager.cc:359] Metadata directory not provided
I20260925 15:23:37.675124 20163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:23:37.676975 20163 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 15:23:37.677842 20178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:37.678159 20163 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:23:37.678253 20163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-2/wal
uuid: "f18797e136f44456af1360609527384d"
format_stamp: "Formatted at 2026-09-25 15:23:16 on dist-test-slave-4bk3"
server_key: "5989802ce37c4cff63c1cee4873e64c1"
server_key_iv: "68a0257ccbc4f193a585eb4e157036b1"
server_key_version: "encryptionkey@0"
I20260925 15:23:37.678668 20163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:23:37.689213 20163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:23:37.691905 20163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:23:37.692302 20163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:23:37.692612 20163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:23:37.692976 20163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:23:37.693033 20163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:37.693076 20163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:23:37.693122 20163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:37.699497 20163 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.195:39309
I20260925 15:23:37.699589 20291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.195:39309 every 8 connection(s)
I20260925 15:23:37.699954 20163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
I20260925 15:23:37.705842 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 20163
Sep 25 15:23:37 dist-test-slave-4bk3 krb5kdc[19253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349817, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.195@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:23:37.712510 20025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:37.700844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:43811 (local address 127.3.211.254:45495)
0925 15:23:37.700968 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:37.700970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:37.701144 (+ 174us) server_negotiation.cc:407] Connection header received
0925 15:23:37.701624 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:37.701625 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:37.701636 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:37.701704 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:37.702702 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:37.703200 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:37.704502 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:37.704752 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:37.704866 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:37.708236 (+ 3370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:37.708237 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:37.708238 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:23:37.708249 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:37.710080 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:37.710966 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:37.710968 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:37.710968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:37.711032 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:37.711389 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:37.711390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:37.711390 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:37.711547 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:37.711615 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:37.712256 (+ 641us) server_negotiation.cc:299] Negotiation successful
0925 15:23:37.712392 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 15:23:37.712653 20294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:37.700804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.195:43811)
0925 15:23:37.700956 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:37.700999 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:37.701499 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:37.701784 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:37.701787 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:37.702357 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:23:37.702556 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:37.702560 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:37.703321 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:37.703322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:37.704210 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:37.704212 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:37.704908 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:37.704909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:37.705051 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:37.705890 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:23:37.705902 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:23:37.708104 (+ 2202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:23:37.710207 (+ 2103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:23:37.710227 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:23:37.710232 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:23:37.710873 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:23:37.711114 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:23:37.711115 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:23:37.711116 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:23:37.711294 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:23:37.711689 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:23:37.711700 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:23:37.712011 (+ 311us) client_negotiation.cc:769] Sending connection context
0925 15:23:37.712160 (+ 149us) client_negotiation.cc:240] Negotiation successful
0925 15:23:37.712408 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":78,"threads_started":1}
I20260925 15:23:37.713271 20292 heartbeater.cc:347] Connected to a master server at 127.3.211.254:45495
I20260925 15:23:37.713416 20292 heartbeater.cc:464] Registering TS with master...
I20260925 15:23:37.713627 20292 heartbeater.cc:511] Master 127.3.211.254:45495 requested a full tablet report, sending...
I20260925 15:23:37.714154 19834 ts_manager.cc:194] Registered new tserver with Master: f18797e136f44456af1360609527384d (127.3.211.195:39309)
I20260925 15:23:37.714661 19834 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.195@KRBTEST.COM'} at 127.3.211.195:43811
I20260925 15:23:37.715873 20296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:37.706367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35822 (local address 127.3.211.254:45495)
0925 15:23:37.706490 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:37.706492 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:37.706658 (+ 166us) server_negotiation.cc:407] Connection header received
0925 15:23:37.706784 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:37.706786 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:37.706815 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:37.706950 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:37.707750 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:37.708385 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:37.709483 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:37.709824 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:37.709921 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:37.712255 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:37.712256 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:37.712257 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:23:37.712264 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:37.714453 (+ 2189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:37.715013 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:37.715014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:37.715015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:37.715072 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:37.715321 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:37.715322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:37.715322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:37.715438 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:37.715633 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:37.715669 (+ 36us) server_negotiation.cc:299] Negotiation successful
0925 15:23:37.715766 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":63,"threads_started":1}
I20260925 15:23:37.716701 3919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:23:37.719965 20296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:37.717147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35834 (local address 127.3.211.254:45495)
0925 15:23:37.717198 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:37.717200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:37.717313 (+ 113us) server_negotiation.cc:407] Connection header received
0925 15:23:37.717406 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:37.717407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:37.717447 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:37.717539 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:23:37.717715 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:37.718244 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:37.719025 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:37.719254 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:37.719334 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:37.719521 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:23:37.719580 (+ 59us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 15:23:37.719885 (+ 305us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35834: token expired
Metrics: {"server-negotiator.queue_time_us":27}
W20260925 15:23:37.720042 20296 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35834: token expired
W20260925 15:23:37.720263 19762 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.3.211.254:45495)
I20260925 15:23:37.728201 20296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:37.720596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35846 (local address 127.3.211.254:45495)
0925 15:23:37.720638 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:37.720640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:37.720648 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:23:37.720738 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:37.720739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:37.720766 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:37.720828 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:37.721348 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:37.721799 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:37.722496 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:37.722697 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:37.722774 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:37.724651 (+ 1877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:37.724653 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:37.724653 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:23:37.724662 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:37.726697 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:37.727243 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:37.727245 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:37.727246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:37.727326 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:37.727565 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:37.727566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:37.727566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:37.727736 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:37.727850 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:37.727986 (+ 136us) server_negotiation.cc:299] Negotiation successful
0925 15:23:37.728081 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 15:23:37.729734 19834 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35846:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:23:37.731331 19834 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:23:37.734130 19953 tablet_service.cc:1467] Processing CreateTablet for tablet 73db2e2724fb45108e91d5e6f9c6838f (DEFAULT_TABLE table=test-table [id=4b2e840b3954498e81de6d641879e0bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:23:37.734340 19953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73db2e2724fb45108e91d5e6f9c6838f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:23:37.735891 19914 tablet_bootstrap.cc:492] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7: Bootstrap starting.
I20260925 15:23:37.736251 19914 tablet_bootstrap.cc:654] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7: Neither blocks nor log segments found. Creating new log.
I20260925 15:23:37.736914 19914 tablet_bootstrap.cc:492] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7: No bootstrap required, opened a new log
I20260925 15:23:37.736959 19914 ts_tablet_manager.cc:1397] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 15:23:37.737113 19914 raft_consensus.cc:348] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f18797e136f44456af1360609527384d" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 39309 } } peers { permanent_uuid: "6746832d79eb409e948e2cd7f1742bfe" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 42381 } } peers { permanent_uuid: "e6429bbd8ad248bca7936b06a1925fc7" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 44505 } }
I20260925 15:23:37.737227 19914 raft_consensus.cc:374] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:23:37.737286 19914 raft_consensus.cc:729] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6429bbd8ad248bca7936b06a1925fc7, State: Initialized, Role: FOLLOWER
I20260925 15:23:37.737334 19914 consensus_queue.cc:260] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f18797e136f44456af1360609527384d" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 39309 } } peers { permanent_uuid: "6746832d79eb409e948e2cd7f1742bfe" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 42381 } } peers { permanent_uuid: "e6429bbd8ad248bca7936b06a1925fc7" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 44505 } }
I20260925 15:23:37.737437 19914 ts_tablet_manager.cc:1428] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 15:23:37.739053 20300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:37.733925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57342 (local address 127.3.211.194:42381)
0925 15:23:37.734135 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:37.734139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:37.734181 (+ 42us) server_negotiation.cc:407] Connection header received
0925 15:23:37.734231 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:37.734234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:37.734256 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:37.734357 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:23:37.734597 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:37.735984 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:37.737566 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:37.738608 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:37.738712 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:37.738785 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:37.738944 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 15:23:37.738981 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":90,"threads_started":1}
I20260925 15:23:37.739110 20298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:37.733643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.194:42381 (local address 127.0.0.1:57342)
0925 15:23:37.733799 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:37.733811 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:37.733863 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:37.734371 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:37.734373 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:37.734375 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:23:37.734437 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:37.734440 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:37.736091 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:37.736093 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:37.737343 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:37.737347 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:37.738756 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:37.738757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:37.738854 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:37.738859 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:23:37.738931 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 15:23:37.739025 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":39,"thread_start_us":97,"threads_started":1}
I20260925 15:23:37.739925 20094 tablet_service.cc:1467] Processing CreateTablet for tablet 73db2e2724fb45108e91d5e6f9c6838f (DEFAULT_TABLE table=test-table [id=4b2e840b3954498e81de6d641879e0bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:23:37.740522 20094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73db2e2724fb45108e91d5e6f9c6838f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:23:37.741572 20299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:37.733786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32982 (local address 127.3.211.195:39309)
0925 15:23:37.736001 (+ 2215us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:37.736007 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:37.736019 (+ 12us) server_negotiation.cc:407] Connection header received
0925 15:23:37.736069 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:37.736071 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:37.736089 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:37.736264 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:23:37.737512 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:37.738763 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:37.740071 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:37.741056 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:37.741186 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:37.741261 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:37.741433 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 15:23:37.741482 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2173,"thread_start_us":117,"threads_started":1}
I20260925 15:23:37.741626 20029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:37.733543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.195:39309 (local address 127.0.0.1:32982)
0925 15:23:37.733621 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:37.733634 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:37.733678 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:37.736231 (+ 2553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:37.736232 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:37.736235 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:23:37.737445 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:37.737447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:37.738856 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:37.738857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:37.739908 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:37.739912 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:37.741197 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:37.741199 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:37.741294 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:37.741300 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 15:23:37.741418 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 15:23:37.741503 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":1162}
I20260925 15:23:37.744987 20226 tablet_service.cc:1467] Processing CreateTablet for tablet 73db2e2724fb45108e91d5e6f9c6838f (DEFAULT_TABLE table=test-table [id=4b2e840b3954498e81de6d641879e0bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:23:37.745344 20226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73db2e2724fb45108e91d5e6f9c6838f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:23:37.748610 20302 tablet_bootstrap.cc:492] T 73db2e2724fb45108e91d5e6f9c6838f P 6746832d79eb409e948e2cd7f1742bfe: Bootstrap starting.
I20260925 15:23:37.748845 20303 tablet_bootstrap.cc:492] T 73db2e2724fb45108e91d5e6f9c6838f P f18797e136f44456af1360609527384d: Bootstrap starting.
I20260925 15:23:37.749447 20303 tablet_bootstrap.cc:654] T 73db2e2724fb45108e91d5e6f9c6838f P f18797e136f44456af1360609527384d: Neither blocks nor log segments found. Creating new log.
I20260925 15:23:37.749475 20302 tablet_bootstrap.cc:654] T 73db2e2724fb45108e91d5e6f9c6838f P 6746832d79eb409e948e2cd7f1742bfe: Neither blocks nor log segments found. Creating new log.
I20260925 15:23:37.750057 20303 log.cc:824] T 73db2e2724fb45108e91d5e6f9c6838f P f18797e136f44456af1360609527384d: Log is configured to *not* fsync() on all Append() calls
I20260925 15:23:37.750339 20302 log.cc:824] T 73db2e2724fb45108e91d5e6f9c6838f P 6746832d79eb409e948e2cd7f1742bfe: Log is configured to *not* fsync() on all Append() calls
I20260925 15:23:37.750967 20303 tablet_bootstrap.cc:492] T 73db2e2724fb45108e91d5e6f9c6838f P f18797e136f44456af1360609527384d: No bootstrap required, opened a new log
I20260925 15:23:37.751050 20303 ts_tablet_manager.cc:1397] T 73db2e2724fb45108e91d5e6f9c6838f P f18797e136f44456af1360609527384d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:23:37.751528 20302 tablet_bootstrap.cc:492] T 73db2e2724fb45108e91d5e6f9c6838f P 6746832d79eb409e948e2cd7f1742bfe: No bootstrap required, opened a new log
I20260925 15:23:37.751609 20302 ts_tablet_manager.cc:1397] T 73db2e2724fb45108e91d5e6f9c6838f P 6746832d79eb409e948e2cd7f1742bfe: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 15:23:37.753227 20303 raft_consensus.cc:348] T 73db2e2724fb45108e91d5e6f9c6838f P f18797e136f44456af1360609527384d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f18797e136f44456af1360609527384d" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 39309 } } peers { permanent_uuid: "6746832d79eb409e948e2cd7f1742bfe" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 42381 } } peers { permanent_uuid: "e6429bbd8ad248bca7936b06a1925fc7" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 44505 } }
I20260925 15:23:37.753456 20303 raft_consensus.cc:374] T 73db2e2724fb45108e91d5e6f9c6838f P f18797e136f44456af1360609527384d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:23:37.753496 20303 raft_consensus.cc:729] T 73db2e2724fb45108e91d5e6f9c6838f P f18797e136f44456af1360609527384d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f18797e136f44456af1360609527384d, State: Initialized, Role: FOLLOWER
I20260925 15:23:37.753650 20303 consensus_queue.cc:260] T 73db2e2724fb45108e91d5e6f9c6838f P f18797e136f44456af1360609527384d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f18797e136f44456af1360609527384d" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 39309 } } peers { permanent_uuid: "6746832d79eb409e948e2cd7f1742bfe" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 42381 } } peers { permanent_uuid: "e6429bbd8ad248bca7936b06a1925fc7" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 44505 } }
I20260925 15:23:37.753705 20302 raft_consensus.cc:348] T 73db2e2724fb45108e91d5e6f9c6838f P 6746832d79eb409e948e2cd7f1742bfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f18797e136f44456af1360609527384d" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 39309 } } peers { permanent_uuid: "6746832d79eb409e948e2cd7f1742bfe" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 42381 } } peers { permanent_uuid: "e6429bbd8ad248bca7936b06a1925fc7" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 44505 } }
I20260925 15:23:37.753912 20302 raft_consensus.cc:374] T 73db2e2724fb45108e91d5e6f9c6838f P 6746832d79eb409e948e2cd7f1742bfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:23:37.753948 20302 raft_consensus.cc:729] T 73db2e2724fb45108e91d5e6f9c6838f P 6746832d79eb409e948e2cd7f1742bfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6746832d79eb409e948e2cd7f1742bfe, State: Initialized, Role: FOLLOWER
I20260925 15:23:37.754006 20303 ts_tablet_manager.cc:1428] T 73db2e2724fb45108e91d5e6f9c6838f P f18797e136f44456af1360609527384d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:23:37.754128 20292 heartbeater.cc:503] Master 127.3.211.254:45495 was elected leader, sending a full tablet report...
I20260925 15:23:37.754097 20302 consensus_queue.cc:260] T 73db2e2724fb45108e91d5e6f9c6838f P 6746832d79eb409e948e2cd7f1742bfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f18797e136f44456af1360609527384d" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 39309 } } peers { permanent_uuid: "6746832d79eb409e948e2cd7f1742bfe" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 42381 } } peers { permanent_uuid: "e6429bbd8ad248bca7936b06a1925fc7" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 44505 } }
I20260925 15:23:37.754396 20302 ts_tablet_manager.cc:1428] T 73db2e2724fb45108e91d5e6f9c6838f P 6746832d79eb409e948e2cd7f1742bfe: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:23:37.754516 20160 heartbeater.cc:503] Master 127.3.211.254:45495 was elected leader, sending a full tablet report...
W20260925 15:23:37.819250 20161 tablet.cc:2369] T 73db2e2724fb45108e91d5e6f9c6838f P 6746832d79eb409e948e2cd7f1742bfe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:23:37.839331 20306 raft_consensus.cc:482] T 73db2e2724fb45108e91d5e6f9c6838f P f18797e136f44456af1360609527384d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:23:37.839463 20306 raft_consensus.cc:504] T 73db2e2724fb45108e91d5e6f9c6838f P f18797e136f44456af1360609527384d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f18797e136f44456af1360609527384d" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 39309 } } peers { permanent_uuid: "6746832d79eb409e948e2cd7f1742bfe" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 42381 } } peers { permanent_uuid: "e6429bbd8ad248bca7936b06a1925fc7" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 44505 } }
I20260925 15:23:37.839803 20306 leader_election.cc:288] T 73db2e2724fb45108e91d5e6f9c6838f P f18797e136f44456af1360609527384d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6746832d79eb409e948e2cd7f1742bfe (127.3.211.194:42381), e6429bbd8ad248bca7936b06a1925fc7 (127.3.211.193:44505)
I20260925 15:23:37.843254 20294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:37.840052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:44505 (local address 127.3.211.195:46823)
0925 15:23:37.840146 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:37.840161 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:37.840254 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:37.840525 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:37.840528 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:37.840531 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:23:37.840659 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:37.840662 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:37.841506 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:37.841507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:37.842355 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:37.842358 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:37.842948 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:37.842949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:37.843037 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:37.843040 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:23:37.843121 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 15:23:37.843168 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 15:23:37.843254 20030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:37.840214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:46823 (local address 127.3.211.193:44505)
0925 15:23:37.840276 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:37.840278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:37.840288 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:23:37.840349 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:37.840350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:37.840361 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:37.840435 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:23:37.840794 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:37.841367 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:37.842462 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:37.842855 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:37.842950 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:37.842959 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:37.843135 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 15:23:37.843185 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 15:23:37.843573 20300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:37.840195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:34333 (local address 127.3.211.194:42381)
0925 15:23:37.840422 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:37.840426 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:37.840515 (+ 89us) server_negotiation.cc:407] Connection header received
0925 15:23:37.840614 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:37.840616 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:37.840633 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:37.840753 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:23:37.841048 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:37.841653 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:37.842795 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:37.843183 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:37.843277 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:37.843287 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:37.843458 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 15:23:37.843508 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 15:23:37.843833 20308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:37.840052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.194:42381 (local address 127.3.211.195:34333)
0925 15:23:37.840428 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:37.840439 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:37.840516 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:37.840796 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:37.840798 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:37.840800 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:23:37.840891 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:37.840893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:37.841760 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:37.841761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:37.842660 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:37.842663 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:37.843276 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:37.843278 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:37.843354 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:37.843358 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:23:37.843707 (+ 349us) client_negotiation.cc:240] Negotiation successful
0925 15:23:37.843753 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":94,"thread_start_us":247,"threads_started":1}
I20260925 15:23:37.843837 19976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73db2e2724fb45108e91d5e6f9c6838f" candidate_uuid: "f18797e136f44456af1360609527384d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6429bbd8ad248bca7936b06a1925fc7" is_pre_election: true
I20260925 15:23:37.844058 19976 raft_consensus.cc:2454] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f18797e136f44456af1360609527384d in term 0.
I20260925 15:23:37.844198 20114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73db2e2724fb45108e91d5e6f9c6838f" candidate_uuid: "f18797e136f44456af1360609527384d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6746832d79eb409e948e2cd7f1742bfe" is_pre_election: true
I20260925 15:23:37.844358 20181 leader_election.cc:302] T 73db2e2724fb45108e91d5e6f9c6838f P f18797e136f44456af1360609527384d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e6429bbd8ad248bca7936b06a1925fc7, f18797e136f44456af1360609527384d; no voters:
I20260925 15:23:37.844386 20114 raft_consensus.cc:2454] T 73db2e2724fb45108e91d5e6f9c6838f P 6746832d79eb409e948e2cd7f1742bfe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f18797e136f44456af1360609527384d in term 0.
I20260925 15:23:37.844512 20306 raft_consensus.cc:2781] T 73db2e2724fb45108e91d5e6f9c6838f P f18797e136f44456af1360609527384d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:23:37.844601 20306 raft_consensus.cc:482] T 73db2e2724fb45108e91d5e6f9c6838f P f18797e136f44456af1360609527384d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:23:37.844630 20306 raft_consensus.cc:3037] T 73db2e2724fb45108e91d5e6f9c6838f P f18797e136f44456af1360609527384d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:23:37.845415 20306 raft_consensus.cc:504] T 73db2e2724fb45108e91d5e6f9c6838f P f18797e136f44456af1360609527384d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f18797e136f44456af1360609527384d" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 39309 } } peers { permanent_uuid: "6746832d79eb409e948e2cd7f1742bfe" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 42381 } } peers { permanent_uuid: "e6429bbd8ad248bca7936b06a1925fc7" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 44505 } }
I20260925 15:23:37.845644 20306 leader_election.cc:288] T 73db2e2724fb45108e91d5e6f9c6838f P f18797e136f44456af1360609527384d [CANDIDATE]: Term 1 election: Requested vote from peers 6746832d79eb409e948e2cd7f1742bfe (127.3.211.194:42381), e6429bbd8ad248bca7936b06a1925fc7 (127.3.211.193:44505)
I20260925 15:23:37.845842 19976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73db2e2724fb45108e91d5e6f9c6838f" candidate_uuid: "f18797e136f44456af1360609527384d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6429bbd8ad248bca7936b06a1925fc7"
I20260925 15:23:37.845909 20114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73db2e2724fb45108e91d5e6f9c6838f" candidate_uuid: "f18797e136f44456af1360609527384d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6746832d79eb409e948e2cd7f1742bfe"
I20260925 15:23:37.845964 19976 raft_consensus.cc:3037] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:23:37.846006 20114 raft_consensus.cc:3037] T 73db2e2724fb45108e91d5e6f9c6838f P 6746832d79eb409e948e2cd7f1742bfe [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:23:37.846740 19976 raft_consensus.cc:2454] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f18797e136f44456af1360609527384d in term 1.
I20260925 15:23:37.846745 20114 raft_consensus.cc:2454] T 73db2e2724fb45108e91d5e6f9c6838f P 6746832d79eb409e948e2cd7f1742bfe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f18797e136f44456af1360609527384d in term 1.
I20260925 15:23:37.846975 20179 leader_election.cc:302] T 73db2e2724fb45108e91d5e6f9c6838f P f18797e136f44456af1360609527384d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6746832d79eb409e948e2cd7f1742bfe, f18797e136f44456af1360609527384d; no voters:
I20260925 15:23:37.847118 20306 raft_consensus.cc:2781] T 73db2e2724fb45108e91d5e6f9c6838f P f18797e136f44456af1360609527384d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:23:37.847268 20306 raft_consensus.cc:686] T 73db2e2724fb45108e91d5e6f9c6838f P f18797e136f44456af1360609527384d [term 1 LEADER]: Becoming Leader. State: Replica: f18797e136f44456af1360609527384d, State: Running, Role: LEADER
I20260925 15:23:37.847343 20306 consensus_queue.cc:237] T 73db2e2724fb45108e91d5e6f9c6838f P f18797e136f44456af1360609527384d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f18797e136f44456af1360609527384d" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 39309 } } peers { permanent_uuid: "6746832d79eb409e948e2cd7f1742bfe" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 42381 } } peers { permanent_uuid: "e6429bbd8ad248bca7936b06a1925fc7" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 44505 } }
I20260925 15:23:37.848315 19833 catalog_manager.cc:5495] T 73db2e2724fb45108e91d5e6f9c6838f P f18797e136f44456af1360609527384d reported cstate change: term changed from 0 to 1, leader changed from <none> to f18797e136f44456af1360609527384d (127.3.211.195). New cstate: current_term: 1 leader_uuid: "f18797e136f44456af1360609527384d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f18797e136f44456af1360609527384d" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 39309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6746832d79eb409e948e2cd7f1742bfe" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 42381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6429bbd8ad248bca7936b06a1925fc7" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 44505 } health_report { overall_health: UNKNOWN } } }
I20260925 15:23:37.858777 20299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:37.856142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32988 (local address 127.3.211.195:39309)
0925 15:23:37.856209 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:37.856211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:37.856225 (+ 14us) server_negotiation.cc:407] Connection header received
0925 15:23:37.856282 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:37.856283 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:37.856301 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:37.856432 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:23:37.856637 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:37.857184 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:37.858044 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:37.858257 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:37.858317 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:37.858482 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:23:37.858582 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:23:37.858662 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:37.858677 (+ 15us) server_negotiation.cc:299] Negotiation successful
0925 15:23:37.858696 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 15:23:37.860734 19976 raft_consensus.cc:1261] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7 [term 1 FOLLOWER]: Refusing update from remote peer f18797e136f44456af1360609527384d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:23:37.860734 20114 raft_consensus.cc:1261] T 73db2e2724fb45108e91d5e6f9c6838f P 6746832d79eb409e948e2cd7f1742bfe [term 1 FOLLOWER]: Refusing update from remote peer f18797e136f44456af1360609527384d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:23:37.861085 20306 consensus_queue.cc:1035] T 73db2e2724fb45108e91d5e6f9c6838f P f18797e136f44456af1360609527384d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6746832d79eb409e948e2cd7f1742bfe" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 42381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:23:37.861253 20309 consensus_queue.cc:1035] T 73db2e2724fb45108e91d5e6f9c6838f P f18797e136f44456af1360609527384d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6429bbd8ad248bca7936b06a1925fc7" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 44505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:23:37.864961 20314 mvcc.cc:205] Tried to move back new op lower bound from 7333272853954461696 to 7333272853903384576. Current Snapshot: MvccSnapshot[applied={T|T < 7333272853954461696}]
I20260925 15:23:37.866526 19834 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:23:37.866655 19834 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:23:37.867449 19834 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07: Sending DeleteTablet for 3 replicas of tablet 73db2e2724fb45108e91d5e6f9c6838f
I20260925 15:23:37.867736 20226 tablet_service.cc:1514] Processing DeleteTablet for tablet 73db2e2724fb45108e91d5e6f9c6838f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:23:37 UTC) from {username='kudu', principal='kudu/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:32982
I20260925 15:23:37.867787 19953 tablet_service.cc:1514] Processing DeleteTablet for tablet 73db2e2724fb45108e91d5e6f9c6838f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:23:37 UTC) from {username='kudu', principal='kudu/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:33690
I20260925 15:23:37.867810 20094 tablet_service.cc:1514] Processing DeleteTablet for tablet 73db2e2724fb45108e91d5e6f9c6838f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:23:37 UTC) from {username='kudu', principal='kudu/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:57342
I20260925 15:23:37.867903 20031 tablet_replica.cc:330] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7: stopping tablet replica
I20260925 15:23:37.867967 20031 raft_consensus.cc:2229] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:23:37.867985 20321 tablet_replica.cc:330] T 73db2e2724fb45108e91d5e6f9c6838f P f18797e136f44456af1360609527384d: stopping tablet replica
I20260925 15:23:37.868031 20031 raft_consensus.cc:2258] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:23:37.868078 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 19872
I20260925 15:23:37.868073 20321 raft_consensus.cc:2229] T 73db2e2724fb45108e91d5e6f9c6838f P f18797e136f44456af1360609527384d [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:23:37.868199 20321 raft_consensus.cc:2258] T 73db2e2724fb45108e91d5e6f9c6838f P f18797e136f44456af1360609527384d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:23:37.869006 20321 ts_tablet_manager.cc:1905] T 73db2e2724fb45108e91d5e6f9c6838f P f18797e136f44456af1360609527384d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:23:37.870772 20321 ts_tablet_manager.cc:1918] T 73db2e2724fb45108e91d5e6f9c6838f P f18797e136f44456af1360609527384d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:23:37.870865 20321 log.cc:1196] T 73db2e2724fb45108e91d5e6f9c6838f P f18797e136f44456af1360609527384d: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-2/wal/wals/73db2e2724fb45108e91d5e6f9c6838f
I20260925 15:23:37.871115 20321 ts_tablet_manager.cc:1939] T 73db2e2724fb45108e91d5e6f9c6838f P f18797e136f44456af1360609527384d: Deleting consensus metadata
I20260925 15:23:37.872130 19820 catalog_manager.cc:4841] TS f18797e136f44456af1360609527384d (127.3.211.195:39309): tablet 73db2e2724fb45108e91d5e6f9c6838f (table test-table [id=4b2e840b3954498e81de6d641879e0bf]) successfully deleted
I20260925 15:23:37.872550 20322 tablet_replica.cc:330] T 73db2e2724fb45108e91d5e6f9c6838f P 6746832d79eb409e948e2cd7f1742bfe: stopping tablet replica
I20260925 15:23:37.872709 20322 raft_consensus.cc:2229] T 73db2e2724fb45108e91d5e6f9c6838f P 6746832d79eb409e948e2cd7f1742bfe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:23:37.872849 20322 raft_consensus.cc:2258] T 73db2e2724fb45108e91d5e6f9c6838f P 6746832d79eb409e948e2cd7f1742bfe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:23:37.873365 20322 ts_tablet_manager.cc:1905] T 73db2e2724fb45108e91d5e6f9c6838f P 6746832d79eb409e948e2cd7f1742bfe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:23:37.875240 20322 ts_tablet_manager.cc:1918] T 73db2e2724fb45108e91d5e6f9c6838f P 6746832d79eb409e948e2cd7f1742bfe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:23:37.875420 20322 log.cc:1196] T 73db2e2724fb45108e91d5e6f9c6838f P 6746832d79eb409e948e2cd7f1742bfe: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-1/wal/wals/73db2e2724fb45108e91d5e6f9c6838f
I20260925 15:23:37.875761 20322 ts_tablet_manager.cc:1939] T 73db2e2724fb45108e91d5e6f9c6838f P 6746832d79eb409e948e2cd7f1742bfe: Deleting consensus metadata
W20260925 15:23:37.876322 19820 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.211.193:44505 (error 108)
I20260925 15:23:37.876482 19818 catalog_manager.cc:4841] TS 6746832d79eb409e948e2cd7f1742bfe (127.3.211.194:42381): tablet 73db2e2724fb45108e91d5e6f9c6838f (table test-table [id=4b2e840b3954498e81de6d641879e0bf]) successfully deleted
I20260925 15:23:37.876909 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 20027
I20260925 15:23:37.877113 20029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:37.876958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:44505 (local address 127.0.0.1:33692)
0925 15:23:37.877022 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:37.877070 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.193:44505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 15:23:37.877681 19820 catalog_manager.cc:4584] TS e6429bbd8ad248bca7936b06a1925fc7 (127.3.211.193:44505): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 73db2e2724fb45108e91d5e6f9c6838f: Network error: Client connection negotiation failed: client connection to 127.3.211.193:44505: connect: Connection refused (error 111)
I20260925 15:23:37.884141 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 20163
I20260925 15:23:37.889456 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 19801
I20260925 15:23:57.895589 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:45495
--webserver_interface=127.3.211.254
--webserver_port=37939
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:41461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.211.254:45495
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:23:57.982991 20324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:23:57.983244 20324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:23:57.983270 20324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:23:57.984577 20324 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 15:23:57.984627 20324 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 15:23:57.984639 20324 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 15:23:57.984651 20324 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:23:57.984664 20324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:23:57.984678 20324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:23:57.984690 20324 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:23:57.984752 20324 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:23:57.986356 20324 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:41461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.211.254:45495
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:45495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.211.254
--webserver_port=37939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:23:57.986680 20324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:23:57.986943 20324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:23:57.992376 20330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:23:57.992401 20324 server_base.cc:1034] running on GCE node
W20260925 15:23:57.992372 20329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:57.992372 20332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:23:57.992935 20324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:23:57.993242 20324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:23:57.994413 20324 hybrid_clock.cc:648] HybridClock initialized: now 1790349837994387 us; error 41 us; skew 500 ppm
Sep 25 15:23:57 dist-test-slave-4bk3 krb5kdc[19253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349837, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:23:57.997023 20324 init.cc:377] Logged in from keytab as kudu/127.3.211.254@KRBTEST.COM (short username kudu)
I20260925 15:23:57.997606 20324 webserver.cc:460] Webserver started at http://127.3.211.254:37939/ using document root <none> and password file <none>
I20260925 15:23:57.997805 20324 fs_manager.cc:359] Metadata directory not provided
I20260925 15:23:57.997849 20324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:23:57.999186 20324 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260925 15:23:57.999866 20339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:58.000044 20324 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:23:58.000159 20324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/master-0/wal
uuid: "486143fcc87440ea915817f8a7a43d07"
format_stamp: "Formatted at 2026-09-25 15:23:16 on dist-test-slave-4bk3"
server_key: "60b243b70c6f0aa2f81924954faba119"
server_key_iv: "b603dc795109e19ca615f908a76b7437"
server_key_version: "encryptionkey@0"
I20260925 15:23:58.000593 20324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:23:58.021925 20324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:23:58.024212 20324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:23:58.024433 20324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:23:58.028647 20324 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.254:45495
I20260925 15:23:58.028704 20391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.254:45495 every 8 connection(s)
I20260925 15:23:58.029102 20324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
I20260925 15:23:58.030531 20392 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:23:58.031150 20392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07: Bootstrap starting.
I20260925 15:23:58.031651 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 20324
I20260925 15:23:58.031973 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.193:44505
--local_ip_for_outbound_sockets=127.3.211.193
--tserver_master_addrs=127.3.211.254:45495
--webserver_port=36767
--webserver_interface=127.3.211.193
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab
--principal=kudu/127.3.211.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:41461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:23:58.040202 20392 log.cc:824] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07: Log is configured to *not* fsync() on all Append() calls
I20260925 15:23:58.042524 20392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:23:58.042896 20392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07: Bootstrap complete.
I20260925 15:23:58.045238 20392 raft_consensus.cc:348] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486143fcc87440ea915817f8a7a43d07" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 45495 } }
I20260925 15:23:58.045536 20392 raft_consensus.cc:729] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 486143fcc87440ea915817f8a7a43d07, State: Initialized, Role: FOLLOWER
I20260925 15:23:58.045729 20392 consensus_queue.cc:260] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486143fcc87440ea915817f8a7a43d07" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 45495 } }
I20260925 15:23:58.045787 20392 raft_consensus.cc:388] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:23:58.045850 20392 raft_consensus.cc:482] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:23:58.045900 20392 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07 [term 2 FOLLOWER]: Advancing to term 3
I20260925 15:23:58.046972 20392 raft_consensus.cc:504] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486143fcc87440ea915817f8a7a43d07" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 45495 } }
I20260925 15:23:58.047063 20392 leader_election.cc:302] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 486143fcc87440ea915817f8a7a43d07; no voters:
I20260925 15:23:58.047292 20392 leader_election.cc:288] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 15:23:58.047333 20396 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 15:23:58.047451 20396 raft_consensus.cc:686] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07 [term 3 LEADER]: Becoming Leader. State: Replica: 486143fcc87440ea915817f8a7a43d07, State: Running, Role: LEADER
I20260925 15:23:58.047540 20396 consensus_queue.cc:237] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486143fcc87440ea915817f8a7a43d07" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 45495 } }
I20260925 15:23:58.047585 20392 sys_catalog.cc:564] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:23:58.047821 20396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 486143fcc87440ea915817f8a7a43d07. Latest consensus state: current_term: 3 leader_uuid: "486143fcc87440ea915817f8a7a43d07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486143fcc87440ea915817f8a7a43d07" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 45495 } } }
I20260925 15:23:58.047888 20396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07 [sys.catalog]: This master's current role is: LEADER
I20260925 15:23:58.047982 20397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "486143fcc87440ea915817f8a7a43d07" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486143fcc87440ea915817f8a7a43d07" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 45495 } } }
I20260925 15:23:58.048105 20400 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:23:58.048141 20397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07 [sys.catalog]: This master's current role is: LEADER
I20260925 15:23:58.049072 20400 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:23:58.049201 20400 catalog_manager.cc:1259] Loaded cluster ID: 9e94e23d535d48fcb7ce57f993218fae
I20260925 15:23:58.049238 20400 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:23:58.050396 20400 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:23:58.050585 20400 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07: Loaded TSK: 0
I20260925 15:23:58.050632 20400 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 486143fcc87440ea915817f8a7a43d07: Loaded TSK: 1
F20260925 15:23:58.050863 20400 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f2cd9ede9ad google::LogMessage::Fail() at ??:0
@ 0x7f2cd9ee188a google::LogMessage::SendToLog() at ??:0
@ 0x7f2cd9ede4de google::LogMessage::Flush() at ??:0
@ 0x7f2cd9edfa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f2cdc64b104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f2cdc648d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f2ce1e381cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f2ce1e3848e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f2ce1e38594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f2ce1dfec48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f2ce1e08ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f2cdbdac0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f2cdbda1954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f2cdcc056db start_thread at ??:0
@ 0x7f2cd89dc61f clone at ??:0
W20260925 15:23:58.136880 20394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:23:58.137101 20394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:23:58.137130 20394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:23:58.138372 20394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:23:58.138419 20394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:23:58.138448 20394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.193
I20260925 15:23:58.140017 20394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:41461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab
--principal=kudu/127.3.211.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.193:44505
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.211.193
--webserver_port=36767
--tserver_master_addrs=127.3.211.254:45495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.193
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:23:58.140327 20394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:23:58.140630 20394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:23:58.145955 20434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:58.145921 20433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:23:58.146186 20394 server_base.cc:1034] running on GCE node
W20260925 15:23:58.145942 20436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:23:58.146497 20394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:23:58.146788 20394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:23:58.147989 20394 hybrid_clock.cc:648] HybridClock initialized: now 1790349838147969 us; error 32 us; skew 500 ppm
Sep 25 15:23:58 dist-test-slave-4bk3 krb5kdc[19253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349838, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:23:58.151208 20394 init.cc:377] Logged in from keytab as kudu/127.3.211.193@KRBTEST.COM (short username kudu)
I20260925 15:23:58.151737 20394 webserver.cc:460] Webserver started at http://127.3.211.193:36767/ using document root <none> and password file <none>
I20260925 15:23:58.151978 20394 fs_manager.cc:359] Metadata directory not provided
I20260925 15:23:58.152050 20394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:23:58.153645 20394 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:23:58.154476 20443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:58.154695 20394 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:23:58.154809 20394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-0/wal
uuid: "e6429bbd8ad248bca7936b06a1925fc7"
format_stamp: "Formatted at 2026-09-25 15:23:16 on dist-test-slave-4bk3"
server_key: "66ffcc9194a142b7a57a5c419ec2e756"
server_key_iv: "2a0ad2d7fcf05d9e92cf94ad90ec9b0d"
server_key_version: "encryptionkey@0"
I20260925 15:23:58.155189 20394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:23:58.177322 20394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:23:58.179791 20394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:23:58.180001 20394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:23:58.180265 20394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:23:58.180797 20450 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 15:23:58.182914 20394 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 15:23:58.183015 20394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 15:23:58.183068 20394 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 15:23:58.183833 20394 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 15:23:58.183897 20394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 15:23:58.183974 20450 tablet_bootstrap.cc:492] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7: Bootstrap starting.
I20260925 15:23:58.191742 20394 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.193:44505
I20260925 15:23:58.191895 20557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.193:44505 every 8 connection(s)
I20260925 15:23:58.192198 20394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:23:58.195176 20450 log.cc:824] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7: Log is configured to *not* fsync() on all Append() calls
I20260925 15:23:58.198042 20450 tablet_bootstrap.cc:492] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:23:58.198156 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 20394
I20260925 15:23:58.198454 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.194:42381
--local_ip_for_outbound_sockets=127.3.211.194
--tserver_master_addrs=127.3.211.254:45495
--webserver_port=34113
--webserver_interface=127.3.211.194
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab
--principal=kudu/127.3.211.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:41461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:23:58.198559 20450 tablet_bootstrap.cc:492] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7: Bootstrap complete.
I20260925 15:23:58.199836 20450 ts_tablet_manager.cc:1397] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.005s
I20260925 15:23:58.201484 20450 raft_consensus.cc:348] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f18797e136f44456af1360609527384d" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 39309 } } peers { permanent_uuid: "6746832d79eb409e948e2cd7f1742bfe" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 42381 } } peers { permanent_uuid: "e6429bbd8ad248bca7936b06a1925fc7" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 44505 } }
I20260925 15:23:58.201848 20450 raft_consensus.cc:729] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6429bbd8ad248bca7936b06a1925fc7, State: Initialized, Role: FOLLOWER
I20260925 15:23:58.202033 20450 consensus_queue.cc:260] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f18797e136f44456af1360609527384d" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 39309 } } peers { permanent_uuid: "6746832d79eb409e948e2cd7f1742bfe" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 42381 } } peers { permanent_uuid: "e6429bbd8ad248bca7936b06a1925fc7" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 44505 } }
I20260925 15:23:58.202518 20450 ts_tablet_manager.cc:1428] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7: Time spent starting tablet: real 0.003s user 0.006s sys 0.000s
W20260925 15:23:58.279001 20560 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:23:58.193021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.193:53797)
0925 15:23:58.193280 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:58.193323 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:58.193805 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:58.278744 (+ 84939us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:45495: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":151,"threads_started":1}
W20260925 15:23:58.279245 20558 heartbeater.cc:644] Failed to heartbeat to 127.3.211.254:45495 (0 consecutive failures): Network error: Failed to ping master at 127.3.211.254:45495: Client connection negotiation failed: client connection to 127.3.211.254:45495: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 15:23:58.279804 20560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:58.279534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.193:40331)
0925 15:23:58.279600 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:58.279651 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 15:23:58.280501 20560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:58.280310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.193:35257)
0925 15:23:58.280418 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:58.280463 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260925 15:23:58.280645 20558 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 15:23:58.287671 20562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:23:58.287950 20562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:23:58.287998 20562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:23:58.289317 20562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:23:58.289393 20562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:23:58.289444 20562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.194
I20260925 15:23:58.291018 20562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:41461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab
--principal=kudu/127.3.211.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.194:42381
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.211.194
--webserver_port=34113
--tserver_master_addrs=127.3.211.254:45495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.194
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:23:58.291321 20562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:23:58.291569 20562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:23:58.297024 20568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:58.297024 20569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:23:58.297104 20562 server_base.cc:1034] running on GCE node
W20260925 15:23:58.297052 20571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:23:58.297619 20562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:23:58.297963 20562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:23:58.299160 20562 hybrid_clock.cc:648] HybridClock initialized: now 1790349838299135 us; error 43 us; skew 500 ppm
Sep 25 15:23:58 dist-test-slave-4bk3 krb5kdc[19253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349838, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:23:58.301811 20562 init.cc:377] Logged in from keytab as kudu/127.3.211.194@KRBTEST.COM (short username kudu)
I20260925 15:23:58.302275 20562 webserver.cc:460] Webserver started at http://127.3.211.194:34113/ using document root <none> and password file <none>
I20260925 15:23:58.302503 20562 fs_manager.cc:359] Metadata directory not provided
I20260925 15:23:58.302556 20562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:23:58.303963 20562 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:23:58.304661 20578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:58.304839 20562 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:23:58.304939 20562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-1/wal
uuid: "6746832d79eb409e948e2cd7f1742bfe"
format_stamp: "Formatted at 2026-09-25 15:23:16 on dist-test-slave-4bk3"
server_key: "2e109b40ef541a9710d704324e500792"
server_key_iv: "8e86b25b90e8c778dde8149850e2b800"
server_key_version: "encryptionkey@0"
I20260925 15:23:58.305313 20562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:23:58.322184 20562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:23:58.324086 20562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:23:58.324263 20562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:23:58.324527 20562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:23:58.324859 20562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:23:58.324893 20562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:58.324915 20562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:23:58.324928 20562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:58.331189 20562 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.194:42381
I20260925 15:23:58.331236 20691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.194:42381 every 8 connection(s)
I20260925 15:23:58.331604 20562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
I20260925 15:23:58.332749 20694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:58.332314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.194:43243)
0925 15:23:58.332519 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:58.332606 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":87,"threads_started":1}
W20260925 15:23:58.333148 20692 heartbeater.cc:644] Failed to heartbeat to 127.3.211.254:45495 (0 consecutive failures): Network error: Failed to ping master at 127.3.211.254:45495: Client connection negotiation failed: client connection to 127.3.211.254:45495: connect: Connection refused (error 111)
I20260925 15:23:58.333457 20694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:58.333361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.194:34473)
0925 15:23:58.333412 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:58.333439 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 15:23:58.333711 20694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:58.333662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.194:38243)
0925 15:23:58.333683 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:58.333700 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260925 15:23:58.333820 20692 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 15:23:58.334887 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 20562
I20260925 15:23:58.335150 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.195:39309
--local_ip_for_outbound_sockets=127.3.211.195
--tserver_master_addrs=127.3.211.254:45495
--webserver_port=39803
--webserver_interface=127.3.211.195
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab
--principal=kudu/127.3.211.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:41461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:23:58.421977 20695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:23:58.422248 20695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:23:58.422289 20695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:23:58.423516 20695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:23:58.423595 20695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:23:58.423641 20695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.195
I20260925 15:23:58.425225 20695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:41461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab
--principal=kudu/127.3.211.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.195:39309
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.211.195
--webserver_port=39803
--tserver_master_addrs=127.3.211.254:45495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.195
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:23:58.425541 20695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:23:58.425793 20695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:23:58.431262 20703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:23:58.431270 20700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:23:58.431638 20695 server_base.cc:1034] running on GCE node
W20260925 15:23:58.431270 20701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:23:58.431957 20695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:23:58.432266 20695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:23:58.433465 20695 hybrid_clock.cc:648] HybridClock initialized: now 1790349838433444 us; error 42 us; skew 500 ppm
Sep 25 15:23:58 dist-test-slave-4bk3 krb5kdc[19253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349838, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:23:58.436232 20695 init.cc:377] Logged in from keytab as kudu/127.3.211.195@KRBTEST.COM (short username kudu)
I20260925 15:23:58.436870 20695 webserver.cc:460] Webserver started at http://127.3.211.195:39803/ using document root <none> and password file <none>
I20260925 15:23:58.437137 20695 fs_manager.cc:359] Metadata directory not provided
I20260925 15:23:58.437213 20695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:23:58.438705 20695 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:23:58.439865 20710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:58.440160 20695 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:23:58.440287 20695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-2/wal
uuid: "f18797e136f44456af1360609527384d"
format_stamp: "Formatted at 2026-09-25 15:23:16 on dist-test-slave-4bk3"
server_key: "5989802ce37c4cff63c1cee4873e64c1"
server_key_iv: "68a0257ccbc4f193a585eb4e157036b1"
server_key_version: "encryptionkey@0"
I20260925 15:23:58.441553 20695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:23:58.467810 20695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:23:58.470117 20695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:23:58.470314 20695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:23:58.470569 20695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:23:58.470889 20695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:23:58.470952 20695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:58.470999 20695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:23:58.471043 20695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:23:58.477304 20695 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.195:39309
I20260925 15:23:58.477352 20823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.195:39309 every 8 connection(s)
I20260925 15:23:58.477727 20695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
I20260925 15:23:58.478850 20826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:58.478426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.195:33519)
0925 15:23:58.478608 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:58.478686 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":75,"threads_started":1}
W20260925 15:23:58.479336 20824 heartbeater.cc:644] Failed to heartbeat to 127.3.211.254:45495 (0 consecutive failures): Network error: Failed to ping master at 127.3.211.254:45495: Client connection negotiation failed: client connection to 127.3.211.254:45495: connect: Connection refused (error 111)
I20260925 15:23:58.479650 20826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:58.479556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.195:36955)
0925 15:23:58.479603 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:58.479631 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 15:23:58.479949 20826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:58.479875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.195:37689)
0925 15:23:58.479915 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:58.479936 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 15:23:58.480073 20824 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 15:23:58.481344 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 20695
I20260925 15:23:59.281355 20828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:59.281112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.193:43571)
0925 15:23:59.281247 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:59.281292 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":64,"threads_started":1}
I20260925 15:23:59.334518 20829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:59.334253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.194:42943)
0925 15:23:59.334399 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:59.334463 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":60,"threads_started":1}
I20260925 15:23:59.480814 20830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:59.480542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.195:55223)
0925 15:23:59.480687 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:59.480746 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":66,"threads_started":1}
I20260925 15:23:59.875207 20831 raft_consensus.cc:482] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:23:59.875330 20831 raft_consensus.cc:504] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f18797e136f44456af1360609527384d" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 39309 } } peers { permanent_uuid: "6746832d79eb409e948e2cd7f1742bfe" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 42381 } } peers { permanent_uuid: "e6429bbd8ad248bca7936b06a1925fc7" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 44505 } }
I20260925 15:23:59.875764 20831 leader_election.cc:288] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f18797e136f44456af1360609527384d (127.3.211.195:39309), 6746832d79eb409e948e2cd7f1742bfe (127.3.211.194:42381)
Sep 25 15:23:59 dist-test-slave-4bk3 krb5kdc[19253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349838, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.193@KRBTEST.COM for kudu/127.3.211.195@KRBTEST.COM
Sep 25 15:23:59 dist-test-slave-4bk3 krb5kdc[19253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349838, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.193@KRBTEST.COM for kudu/127.3.211.194@KRBTEST.COM
I20260925 15:23:59.887028 20833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:59.875994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.194:42381 (local address 127.3.211.193:52575)
0925 15:23:59.876146 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:59.876157 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:59.876216 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:59.877043 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:59.877046 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:59.877434 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:23:59.877627 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:59.877633 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:59.878764 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:59.878764 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:59.879543 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:59.879545 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:59.880159 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:59.880161 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:59.880290 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:59.880949 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:23:59.880966 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:23:59.883299 (+ 2333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:23:59.885420 (+ 2121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:23:59.885420 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:23:59.885421 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:23:59.885632 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:23:59.885920 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:23:59.885921 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:23:59.885922 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:23:59.886049 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:23:59.886469 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:23:59.886476 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:23:59.886613 (+ 137us) client_negotiation.cc:769] Sending connection context
0925 15:23:59.886726 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 15:23:59.886931 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":58,"thread_start_us":84,"threads_started":1}
I20260925 15:23:59.887042 20835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:59.876098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:35403 (local address 127.3.211.195:39309)
0925 15:23:59.876250 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:59.876262 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:59.876295 (+ 33us) server_negotiation.cc:407] Connection header received
0925 15:23:59.876792 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:59.876795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:59.876836 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:59.876998 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:59.877779 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:59.878624 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:59.879646 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:59.880054 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:59.880184 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:59.882902 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:59.882907 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:59.882907 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:23:59.882919 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:59.885136 (+ 2217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:59.885718 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:59.885728 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:59.885728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:59.885839 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:59.886143 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:59.886144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:59.886144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:59.886391 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:59.886489 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:59.886789 (+ 300us) server_negotiation.cc:299] Negotiation successful
0925 15:23:59.886919 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":55,"threads_started":1}
I20260925 15:23:59.887059 20834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:59.876085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:52575 (local address 127.3.211.194:42381)
0925 15:23:59.876274 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 15:23:59.876282 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:23:59.876300 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:23:59.876791 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:23:59.876793 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:23:59.876836 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:23:59.876997 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:23:59.877761 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:59.878630 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:59.879641 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:23:59.880057 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:23:59.880181 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:59.883422 (+ 3241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:23:59.883427 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:23:59.883427 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:23:59.883441 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:23:59.885323 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:59.885746 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:59.885749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:59.885749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:59.885838 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:23:59.886130 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:23:59.886131 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:23:59.886131 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:23:59.886391 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:23:59.886489 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 15:23:59.886781 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 15:23:59.886923 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":46,"threads_started":1}
I20260925 15:23:59.887379 20832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:23:59.875924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.195:39309 (local address 127.3.211.193:35403)
0925 15:23:59.876069 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 15:23:59.876081 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:23:59.876143 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:23:59.877044 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:23:59.877046 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:23:59.877433 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:23:59.877627 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:59.877633 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:59.878736 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:59.878737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:59.879542 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:23:59.879544 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:23:59.880162 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:23:59.880163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:23:59.880285 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:23:59.880987 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:23:59.881004 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:23:59.882738 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:23:59.885248 (+ 2510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:23:59.885265 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:23:59.885268 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:23:59.885618 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:23:59.885917 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:23:59.885918 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:23:59.885919 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:23:59.886046 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:23:59.886470 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:23:59.886477 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:23:59.886613 (+ 136us) client_negotiation.cc:769] Sending connection context
0925 15:23:59.886725 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 15:23:59.887011 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":145,"thread_start_us":65,"threads_started":1}
W20260925 15:23:59.888180 20444 leader_election.cc:341] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6746832d79eb409e948e2cd7f1742bfe (127.3.211.194:42381): Not found: Tablet not found: 73db2e2724fb45108e91d5e6f9c6838f
W20260925 15:23:59.888273 20446 leader_election.cc:341] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f18797e136f44456af1360609527384d (127.3.211.195:39309): Not found: Tablet not found: 73db2e2724fb45108e91d5e6f9c6838f
I20260925 15:23:59.888388 20446 leader_election.cc:302] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6429bbd8ad248bca7936b06a1925fc7; no voters: 6746832d79eb409e948e2cd7f1742bfe, f18797e136f44456af1360609527384d
I20260925 15:23:59.888525 20831 raft_consensus.cc:2726] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:24:00.282120 20832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:00.281932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.193:34113)
0925 15:24:00.282030 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:00.282072 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260925 15:24:00.335322 20836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:00.335077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.194:53323)
0925 15:24:00.335220 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:00.335271 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":70,"threads_started":1}
I20260925 15:24:00.481936 20837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:00.481661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.195:54453)
0925 15:24:00.481843 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:00.481891 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":77,"threads_started":1}
I20260925 15:24:01.282972 20838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:01.282742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.193:60499)
0925 15:24:01.282880 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:01.282926 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":70,"threads_started":1}
I20260925 15:24:01.336179 20839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:01.335943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.194:41225)
0925 15:24:01.336061 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:01.336117 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":57,"threads_started":1}
I20260925 15:24:01.482754 20840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:01.482489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.195:47275)
0925 15:24:01.482656 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:01.482707 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":69,"threads_started":1}
I20260925 15:24:01.641772 20841 raft_consensus.cc:482] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:24:01.641950 20841 raft_consensus.cc:504] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f18797e136f44456af1360609527384d" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 39309 } } peers { permanent_uuid: "6746832d79eb409e948e2cd7f1742bfe" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 42381 } } peers { permanent_uuid: "e6429bbd8ad248bca7936b06a1925fc7" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 44505 } }
I20260925 15:24:01.642323 20841 leader_election.cc:288] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f18797e136f44456af1360609527384d (127.3.211.195:39309), 6746832d79eb409e948e2cd7f1742bfe (127.3.211.194:42381)
W20260925 15:24:01.642875 20446 leader_election.cc:341] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f18797e136f44456af1360609527384d (127.3.211.195:39309): Not found: Tablet not found: 73db2e2724fb45108e91d5e6f9c6838f
W20260925 15:24:01.642970 20444 leader_election.cc:341] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6746832d79eb409e948e2cd7f1742bfe (127.3.211.194:42381): Not found: Tablet not found: 73db2e2724fb45108e91d5e6f9c6838f
I20260925 15:24:01.643047 20444 leader_election.cc:302] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6429bbd8ad248bca7936b06a1925fc7; no voters: 6746832d79eb409e948e2cd7f1742bfe, f18797e136f44456af1360609527384d
I20260925 15:24:01.643133 20841 raft_consensus.cc:2726] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:24:02.284014 20842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:02.283727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.193:45833)
0925 15:24:02.283919 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:02.283968 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":90,"threads_started":1}
I20260925 15:24:02.337040 20843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:02.336785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.194:50487)
0925 15:24:02.336947 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:02.336993 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":80,"threads_started":1}
I20260925 15:24:02.483552 20844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:02.483334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.195:47029)
0925 15:24:02.483463 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:02.483507 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":63,"threads_started":1}
I20260925 15:24:03.285076 20845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:03.284807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.193:44435)
0925 15:24:03.284966 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:03.285001 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":84,"threads_started":1}
I20260925 15:24:03.337955 20846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:03.337594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.194:36299)
0925 15:24:03.337812 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:03.337888 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":63,"threads_started":1}
I20260925 15:24:03.484472 20847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:03.484159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.195:40515)
0925 15:24:03.484319 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:03.484423 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":65,"threads_started":1}
I20260925 15:24:04.285833 20848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:04.285587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.193:35919)
0925 15:24:04.285743 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:04.285786 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":64,"threads_started":1}
I20260925 15:24:04.339046 20849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:04.338657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.194:55935)
0925 15:24:04.338892 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:04.338961 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":103,"threads_started":1}
I20260925 15:24:04.485451 20850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:04.485205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.195:39225)
0925 15:24:04.485357 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:04.485403 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":63,"threads_started":1}
I20260925 15:24:04.671464 20851 raft_consensus.cc:482] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:24:04.671561 20851 raft_consensus.cc:504] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f18797e136f44456af1360609527384d" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 39309 } } peers { permanent_uuid: "6746832d79eb409e948e2cd7f1742bfe" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 42381 } } peers { permanent_uuid: "e6429bbd8ad248bca7936b06a1925fc7" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 44505 } }
I20260925 15:24:04.671828 20851 leader_election.cc:288] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f18797e136f44456af1360609527384d (127.3.211.195:39309), 6746832d79eb409e948e2cd7f1742bfe (127.3.211.194:42381)
W20260925 15:24:04.672161 20446 leader_election.cc:341] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f18797e136f44456af1360609527384d (127.3.211.195:39309): Not found: Tablet not found: 73db2e2724fb45108e91d5e6f9c6838f
W20260925 15:24:04.672232 20444 leader_election.cc:341] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6746832d79eb409e948e2cd7f1742bfe (127.3.211.194:42381): Not found: Tablet not found: 73db2e2724fb45108e91d5e6f9c6838f
I20260925 15:24:04.672327 20444 leader_election.cc:302] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6429bbd8ad248bca7936b06a1925fc7; no voters: 6746832d79eb409e948e2cd7f1742bfe, f18797e136f44456af1360609527384d
I20260925 15:24:04.672423 20851 raft_consensus.cc:2726] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:24:05.286746 20852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:05.286479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.193:48427)
0925 15:24:05.286648 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:05.286698 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":68,"threads_started":1}
I20260925 15:24:05.339901 20853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:05.339640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.194:36229)
0925 15:24:05.339809 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:05.339852 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":76,"threads_started":1}
I20260925 15:24:05.486349 20854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:05.486110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.195:52283)
0925 15:24:05.486258 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:05.486301 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":66,"threads_started":1}
I20260925 15:24:06.288120 20855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:06.287724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.193:34225)
0925 15:24:06.287984 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:06.288053 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":133,"threads_started":1}
I20260925 15:24:06.340919 20856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:06.340626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.194:46161)
0925 15:24:06.340803 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:06.340856 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":111,"threads_started":1}
I20260925 15:24:06.487280 20857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:06.487005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.195:52005)
0925 15:24:06.487186 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:06.487232 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":85,"threads_started":1}
I20260925 15:24:07.289098 20858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:07.288823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.193:53077)
0925 15:24:07.288997 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:07.289045 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":71,"threads_started":1}
I20260925 15:24:07.341727 20859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:07.341477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.194:33897)
0925 15:24:07.341634 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:07.341680 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":66,"threads_started":1}
I20260925 15:24:07.488162 20860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:07.487904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.195:56785)
0925 15:24:07.488064 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:07.488111 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":66,"threads_started":1}
I20260925 15:24:08.290380 20861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:08.289991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.193:56017)
0925 15:24:08.290255 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:08.290320 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":117,"threads_started":1}
I20260925 15:24:08.343003 20862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:08.342620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.194:48671)
0925 15:24:08.342865 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:08.342936 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":107,"threads_started":1}
I20260925 15:24:08.434804 20863 raft_consensus.cc:482] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:24:08.434916 20863 raft_consensus.cc:504] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f18797e136f44456af1360609527384d" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 39309 } } peers { permanent_uuid: "6746832d79eb409e948e2cd7f1742bfe" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 42381 } } peers { permanent_uuid: "e6429bbd8ad248bca7936b06a1925fc7" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 44505 } }
I20260925 15:24:08.435173 20863 leader_election.cc:288] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f18797e136f44456af1360609527384d (127.3.211.195:39309), 6746832d79eb409e948e2cd7f1742bfe (127.3.211.194:42381)
W20260925 15:24:08.435797 20444 leader_election.cc:341] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6746832d79eb409e948e2cd7f1742bfe (127.3.211.194:42381): Not found: Tablet not found: 73db2e2724fb45108e91d5e6f9c6838f
W20260925 15:24:08.435880 20446 leader_election.cc:341] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f18797e136f44456af1360609527384d (127.3.211.195:39309): Not found: Tablet not found: 73db2e2724fb45108e91d5e6f9c6838f
I20260925 15:24:08.435927 20446 leader_election.cc:302] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6429bbd8ad248bca7936b06a1925fc7; no voters: 6746832d79eb409e948e2cd7f1742bfe, f18797e136f44456af1360609527384d
I20260925 15:24:08.436039 20863 raft_consensus.cc:2726] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:24:08.489068 20864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:08.488806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.195:53101)
0925 15:24:08.488979 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:08.489015 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":126,"threads_started":1}
I20260925 15:24:09.291481 20865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:09.291224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.193:48101)
0925 15:24:09.291376 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:09.291424 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":69,"threads_started":1}
I20260925 15:24:09.344023 20866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:09.343737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.194:37873)
0925 15:24:09.343915 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:09.343965 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":111,"threads_started":1}
I20260925 15:24:09.489797 20867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:09.489577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.195:50981)
0925 15:24:09.489711 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:09.489750 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":72,"threads_started":1}
I20260925 15:24:10.292558 20868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:10.292205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.193:44233)
0925 15:24:10.292464 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:10.292512 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":68,"threads_started":1}
I20260925 15:24:10.345010 20869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:10.344696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.194:36597)
0925 15:24:10.344920 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:10.344964 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":61,"threads_started":1}
I20260925 15:24:10.490581 20870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:10.490344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.195:56859)
0925 15:24:10.490479 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:10.490522 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":59,"threads_started":1}
I20260925 15:24:10.722661 20871 raft_consensus.cc:482] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:24:10.722771 20871 raft_consensus.cc:504] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f18797e136f44456af1360609527384d" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 39309 } } peers { permanent_uuid: "6746832d79eb409e948e2cd7f1742bfe" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 42381 } } peers { permanent_uuid: "e6429bbd8ad248bca7936b06a1925fc7" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 44505 } }
I20260925 15:24:10.723012 20871 leader_election.cc:288] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f18797e136f44456af1360609527384d (127.3.211.195:39309), 6746832d79eb409e948e2cd7f1742bfe (127.3.211.194:42381)
W20260925 15:24:10.723519 20444 leader_election.cc:341] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6746832d79eb409e948e2cd7f1742bfe (127.3.211.194:42381): Not found: Tablet not found: 73db2e2724fb45108e91d5e6f9c6838f
W20260925 15:24:10.723752 20446 leader_election.cc:341] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f18797e136f44456af1360609527384d (127.3.211.195:39309): Not found: Tablet not found: 73db2e2724fb45108e91d5e6f9c6838f
I20260925 15:24:10.723825 20446 leader_election.cc:302] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6429bbd8ad248bca7936b06a1925fc7; no voters: 6746832d79eb409e948e2cd7f1742bfe, f18797e136f44456af1360609527384d
I20260925 15:24:10.723945 20871 raft_consensus.cc:2726] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:24:11.293322 20872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:11.293075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.193:59737)
0925 15:24:11.293229 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:11.293267 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":66,"threads_started":1}
I20260925 15:24:11.345871 20873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:11.345631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.194:48017)
0925 15:24:11.345775 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:11.345811 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":74,"threads_started":1}
I20260925 15:24:11.491474 20874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:11.491215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.195:48645)
0925 15:24:11.491381 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:11.491428 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":70,"threads_started":1}
I20260925 15:24:12.294736 20875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:12.293969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.193:36197)
0925 15:24:12.294581 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:12.294665 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":138,"threads_started":1}
I20260925 15:24:12.346661 20876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:12.346356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.194:42493)
0925 15:24:12.346514 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:12.346613 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":66,"threads_started":1}
I20260925 15:24:12.492391 20877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:12.492063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.195:35843)
0925 15:24:12.492228 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:12.492277 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":65,"threads_started":1}
I20260925 15:24:13.295733 20878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:13.295497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.193:56169)
0925 15:24:13.295641 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:13.295685 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":59,"threads_started":1}
I20260925 15:24:13.347790 20879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:13.347528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.194:41343)
0925 15:24:13.347691 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:13.347737 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":69,"threads_started":1}
I20260925 15:24:13.453081 20880 raft_consensus.cc:482] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:24:13.453190 20880 raft_consensus.cc:504] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f18797e136f44456af1360609527384d" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 39309 } } peers { permanent_uuid: "6746832d79eb409e948e2cd7f1742bfe" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 42381 } } peers { permanent_uuid: "e6429bbd8ad248bca7936b06a1925fc7" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 44505 } }
I20260925 15:24:13.453423 20880 leader_election.cc:288] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f18797e136f44456af1360609527384d (127.3.211.195:39309), 6746832d79eb409e948e2cd7f1742bfe (127.3.211.194:42381)
W20260925 15:24:13.453889 20446 leader_election.cc:341] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f18797e136f44456af1360609527384d (127.3.211.195:39309): Not found: Tablet not found: 73db2e2724fb45108e91d5e6f9c6838f
W20260925 15:24:13.453979 20444 leader_election.cc:341] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6746832d79eb409e948e2cd7f1742bfe (127.3.211.194:42381): Not found: Tablet not found: 73db2e2724fb45108e91d5e6f9c6838f
I20260925 15:24:13.454058 20444 leader_election.cc:302] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6429bbd8ad248bca7936b06a1925fc7; no voters: 6746832d79eb409e948e2cd7f1742bfe, f18797e136f44456af1360609527384d
I20260925 15:24:13.454159 20880 raft_consensus.cc:2726] T 73db2e2724fb45108e91d5e6f9c6838f P e6429bbd8ad248bca7936b06a1925fc7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:24:13.493399 20881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:13.493129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:45495 (local address 127.3.211.195:46353)
0925 15:24:13.493285 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:13.493326 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.254:45495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":69,"threads_started":1}
I20260925 15:24:13.493381 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 20394
I20260925 15:24:13.499161 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 20562
I20260925 15:24:13.505317 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 20695
2026-09-25T15:24:13Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59285 ms)
[----------] 30 tests from SecurityITest (243742 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:24:14 dist-test-slave-4bk3 krb5kdc[20884](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:24:14 dist-test-slave-4bk3 krb5kdc[20884](info): set up 2 sockets
Sep 25 15:24:14 dist-test-slave-4bk3 krb5kdc[20884](info): commencing operation
krb5kdc: starting...
W20260925 15:24:15.416834 3919 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Sep 25 15:24:15 dist-test-slave-4bk3 krb5kdc[20884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:24:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:24:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.211.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
I20260925 15:24:15.553387 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:43173
--webserver_interface=127.3.211.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:36559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.211.254:43173
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:24:15.646494 20897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:15.646754 20897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:15.646780 20897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:15.648048 20897 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:24:15.648108 20897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:15.648123 20897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:15.648136 20897 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:24:15.648149 20897 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:24:15.649840 20897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:36559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.211.254:43173
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:43173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.211.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:24:15.650102 20897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:15.650301 20897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:15.656268 20902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:15.656410 20903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:15.656522 20897 server_base.cc:1034] running on GCE node
W20260925 15:24:15.656420 20905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:15.656927 20897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:15.657274 20897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:15.658458 20897 hybrid_clock.cc:648] HybridClock initialized: now 1790349855658423 us; error 51 us; skew 500 ppm
Sep 25 15:24:15 dist-test-slave-4bk3 krb5kdc[20884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349855, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:24:15.661684 20897 init.cc:377] Logged in from keytab as kudu/127.3.211.254@KRBTEST.COM (short username kudu)
I20260925 15:24:15.662236 20897 webserver.cc:460] Webserver started at http://127.3.211.254:41739/ using document root <none> and password file <none>
I20260925 15:24:15.662607 20897 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:15.662705 20897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:15.662875 20897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:15.664129 20897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/master-0/data/instance:
uuid: "813298235fc34d168e4f56b77dd5361c"
format_stamp: "Formatted at 2026-09-25 15:24:15 on dist-test-slave-4bk3"
server_key: "28bd88817e9f76b4581ecee8a47d94d4"
server_key_iv: "f4fe77c816ef96e540d269ae993c533a"
server_key_version: "encryptionkey@0"
I20260925 15:24:15.664618 20897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/master-0/wal/instance:
uuid: "813298235fc34d168e4f56b77dd5361c"
format_stamp: "Formatted at 2026-09-25 15:24:15 on dist-test-slave-4bk3"
server_key: "28bd88817e9f76b4581ecee8a47d94d4"
server_key_iv: "f4fe77c816ef96e540d269ae993c533a"
server_key_version: "encryptionkey@0"
I20260925 15:24:15.666494 20897 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260925 15:24:15.667663 20912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:15.668108 20897 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:24:15.668260 20897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/master-0/wal
uuid: "813298235fc34d168e4f56b77dd5361c"
format_stamp: "Formatted at 2026-09-25 15:24:15 on dist-test-slave-4bk3"
server_key: "28bd88817e9f76b4581ecee8a47d94d4"
server_key_iv: "f4fe77c816ef96e540d269ae993c533a"
server_key_version: "encryptionkey@0"
I20260925 15:24:15.668407 20897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:15.681015 20897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:15.683856 20897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:15.684077 20897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:15.689985 20897 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.254:43173
I20260925 15:24:15.690080 20964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.254:43173 every 8 connection(s)
I20260925 15:24:15.690685 20897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
I20260925 15:24:15.691979 20965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:15.694819 20965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 813298235fc34d168e4f56b77dd5361c: Bootstrap starting.
I20260925 15:24:15.695432 20965 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 813298235fc34d168e4f56b77dd5361c: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:15.695688 20965 log.cc:824] T 00000000000000000000000000000000 P 813298235fc34d168e4f56b77dd5361c: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:15.696640 20965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 813298235fc34d168e4f56b77dd5361c: No bootstrap required, opened a new log
I20260925 15:24:15.698345 20965 raft_consensus.cc:348] T 00000000000000000000000000000000 P 813298235fc34d168e4f56b77dd5361c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "813298235fc34d168e4f56b77dd5361c" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 43173 } }
I20260925 15:24:15.698506 20965 raft_consensus.cc:374] T 00000000000000000000000000000000 P 813298235fc34d168e4f56b77dd5361c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:24:15.698539 20965 raft_consensus.cc:729] T 00000000000000000000000000000000 P 813298235fc34d168e4f56b77dd5361c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 813298235fc34d168e4f56b77dd5361c, State: Initialized, Role: FOLLOWER
I20260925 15:24:15.698696 20965 consensus_queue.cc:260] T 00000000000000000000000000000000 P 813298235fc34d168e4f56b77dd5361c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "813298235fc34d168e4f56b77dd5361c" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 43173 } }
I20260925 15:24:15.698786 20965 raft_consensus.cc:388] T 00000000000000000000000000000000 P 813298235fc34d168e4f56b77dd5361c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:24:15.698817 20965 raft_consensus.cc:482] T 00000000000000000000000000000000 P 813298235fc34d168e4f56b77dd5361c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:24:15.698848 20965 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 813298235fc34d168e4f56b77dd5361c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:15.699440 20965 raft_consensus.cc:504] T 00000000000000000000000000000000 P 813298235fc34d168e4f56b77dd5361c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "813298235fc34d168e4f56b77dd5361c" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 43173 } }
I20260925 15:24:15.699543 20965 leader_election.cc:302] T 00000000000000000000000000000000 P 813298235fc34d168e4f56b77dd5361c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 813298235fc34d168e4f56b77dd5361c; no voters:
I20260925 15:24:15.699721 20965 leader_election.cc:288] T 00000000000000000000000000000000 P 813298235fc34d168e4f56b77dd5361c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:24:15.699760 20968 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 813298235fc34d168e4f56b77dd5361c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:24:15.699880 20968 raft_consensus.cc:686] T 00000000000000000000000000000000 P 813298235fc34d168e4f56b77dd5361c [term 1 LEADER]: Becoming Leader. State: Replica: 813298235fc34d168e4f56b77dd5361c, State: Running, Role: LEADER
I20260925 15:24:15.699954 20968 consensus_queue.cc:237] T 00000000000000000000000000000000 P 813298235fc34d168e4f56b77dd5361c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "813298235fc34d168e4f56b77dd5361c" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 43173 } }
I20260925 15:24:15.700017 20965 sys_catalog.cc:564] T 00000000000000000000000000000000 P 813298235fc34d168e4f56b77dd5361c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:24:15.700428 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 20897
I20260925 15:24:15.700555 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/master-0/wal/instance
I20260925 15:24:15.700731 3919 external_mini_cluster.cc:1428] Setting key 0297a2ab54b55c9e7234e4c28e57befe
I20260925 15:24:15.701428 20969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 813298235fc34d168e4f56b77dd5361c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "813298235fc34d168e4f56b77dd5361c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "813298235fc34d168e4f56b77dd5361c" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 43173 } } }
I20260925 15:24:15.701535 20969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 813298235fc34d168e4f56b77dd5361c [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:15.701743 20982 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:24:15.701833 20970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 813298235fc34d168e4f56b77dd5361c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 813298235fc34d168e4f56b77dd5361c. Latest consensus state: current_term: 1 leader_uuid: "813298235fc34d168e4f56b77dd5361c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "813298235fc34d168e4f56b77dd5361c" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 43173 } } }
I20260925 15:24:15.702204 20970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 813298235fc34d168e4f56b77dd5361c [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:15.702482 20982 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:24:15.704806 20982 catalog_manager.cc:1347] Generated new cluster ID: ed80e5fa384d48dab240ce3fc34646f9
I20260925 15:24:15.704895 20982 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 15:24:15 dist-test-slave-4bk3 krb5kdc[20884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:24:15.714855 20983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:15.701531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37978 (local address 127.3.211.254:43173)
0925 15:24:15.701730 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:15.701741 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:15.701778 (+ 37us) server_negotiation.cc:407] Connection header received
0925 15:24:15.702000 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:15.702007 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:15.702069 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:15.703118 (+ 1049us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:15.703222 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:15.704307 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:15.705474 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:15.706366 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:15.706548 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:15.710429 (+ 3881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:15.710436 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:15.710437 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:15.710454 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:15.712736 (+ 2282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:15.713210 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:15.713215 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:15.713215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:15.713357 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:15.713596 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:15.713597 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:15.713597 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:15.713955 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:15.714131 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:15.714249 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 15:24:15.714615 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":95,"threads_started":1}
I20260925 15:24:15.714857 20982 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:24:15.715595 20982 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:24:15.718714 20982 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 813298235fc34d168e4f56b77dd5361c: Generated new TSK 0
I20260925 15:24:15.718935 20982 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.211.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab.
Entry for principal kudu/127.3.211.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab.
I20260925 15:24:15.749514 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.193:0
--local_ip_for_outbound_sockets=127.3.211.193
--webserver_interface=127.3.211.193
--webserver_port=0
--tserver_master_addrs=127.3.211.254:43173
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab
--principal=kudu/127.3.211.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:36559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:24:15.839480 20991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:15.839706 20991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:15.839735 20991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:15.842589 20991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:15.842679 20991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:15.842720 20991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.193
I20260925 15:24:15.844887 20991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:36559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab
--principal=kudu/127.3.211.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.211.193
--webserver_port=0
--tserver_master_addrs=127.3.211.254:43173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.193
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:24:15.845165 20991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:15.845444 20991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:15.850713 20996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:15.850713 20997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:15.850908 20991 server_base.cc:1034] running on GCE node
W20260925 15:24:15.850708 20999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:15.851330 20991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:15.851826 20991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:15.853156 20991 hybrid_clock.cc:648] HybridClock initialized: now 1790349855853108 us; error 69 us; skew 500 ppm
Sep 25 15:24:15 dist-test-slave-4bk3 krb5kdc[20884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349855, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:24:15.856822 20991 init.cc:377] Logged in from keytab as kudu/127.3.211.193@KRBTEST.COM (short username kudu)
I20260925 15:24:15.857385 20991 webserver.cc:460] Webserver started at http://127.3.211.193:45633/ using document root <none> and password file <none>
I20260925 15:24:15.857733 20991 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:15.857838 20991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:15.858035 20991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:15.859213 20991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/ts-0/data/instance:
uuid: "e37e97b661f04f24ac14a7101ebaf400"
format_stamp: "Formatted at 2026-09-25 15:24:15 on dist-test-slave-4bk3"
server_key: "031e5db6612a0dd4fc11e37d531e56f8"
server_key_iv: "f26e807ccdbecef026b47dcf2b09de5f"
server_key_version: "encryptionkey@0"
I20260925 15:24:15.859649 20991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/ts-0/wal/instance:
uuid: "e37e97b661f04f24ac14a7101ebaf400"
format_stamp: "Formatted at 2026-09-25 15:24:15 on dist-test-slave-4bk3"
server_key: "031e5db6612a0dd4fc11e37d531e56f8"
server_key_iv: "f26e807ccdbecef026b47dcf2b09de5f"
server_key_version: "encryptionkey@0"
I20260925 15:24:15.861325 20991 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260925 15:24:15.862468 21006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:15.862809 20991 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:24:15.862947 20991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/ts-0/wal
uuid: "e37e97b661f04f24ac14a7101ebaf400"
format_stamp: "Formatted at 2026-09-25 15:24:15 on dist-test-slave-4bk3"
server_key: "031e5db6612a0dd4fc11e37d531e56f8"
server_key_iv: "f26e807ccdbecef026b47dcf2b09de5f"
server_key_version: "encryptionkey@0"
I20260925 15:24:15.863041 20991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:15.883445 20991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:15.886068 20991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:15.886264 20991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:15.886519 20991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:24:15.886849 20991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:24:15.886907 20991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:15.886943 20991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:24:15.886991 20991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:15.893219 20991 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.193:44327
I20260925 15:24:15.893422 21119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.193:44327 every 8 connection(s)
I20260925 15:24:15.893834 20991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:24:15.896185 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 20991
I20260925 15:24:15.896546 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/ts-0/wal/instance
I20260925 15:24:15.896993 3919 external_mini_cluster.cc:1428] Setting key 2934779c4b0027fed63bc95779347cd2
Sep 25 15:24:15 dist-test-slave-4bk3 krb5kdc[20884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349855, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.193@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:24:15.906049 21122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:15.894668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:43173 (local address 127.3.211.193:57907)
0925 15:24:15.894928 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:15.894979 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:15.895528 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:15.895798 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:15.895802 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:15.896523 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:24:15.896747 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:15.896752 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:15.897671 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:15.897672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:15.898539 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:15.898542 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:15.898965 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:15.898966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:15.899097 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:15.899685 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:24:15.899693 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:24:15.901539 (+ 1846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:24:15.903866 (+ 2327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:15.903883 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:15.903886 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:15.904320 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:15.904823 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:15.904825 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:15.904826 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:15.905022 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:15.905397 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:24:15.905407 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:24:15.905585 (+ 178us) client_negotiation.cc:769] Sending connection context
0925 15:24:15.905688 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 15:24:15.905848 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":108,"threads_started":1}
I20260925 15:24:15.906064 20983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:15.894789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:57907 (local address 127.3.211.254:43173)
0925 15:24:15.894904 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:15.894907 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:15.895085 (+ 178us) server_negotiation.cc:407] Connection header received
0925 15:24:15.895645 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:15.895647 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:15.895664 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:15.895724 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:15.896909 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:15.897575 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:15.898642 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:15.898864 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:15.898955 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:15.901681 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:15.901683 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:15.901683 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:15.901693 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:15.903694 (+ 2001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:15.904569 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:15.904571 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:15.904572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:15.904658 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:15.905123 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:15.905124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:15.905125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:15.905295 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:15.905377 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:15.905856 (+ 479us) server_negotiation.cc:299] Negotiation successful
0925 15:24:15.905960 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 15:24:15.906891 21120 heartbeater.cc:347] Connected to a master server at 127.3.211.254:43173
WARNING: no policy specified for kudu/127.3.211.194@KRBTEST.COM; defaulting to no policy
I20260925 15:24:15.907042 21120 heartbeater.cc:464] Registering TS with master...
I20260925 15:24:15.907269 21120 heartbeater.cc:511] Master 127.3.211.254:43173 requested a full tablet report, sending...
I20260925 15:24:15.907809 20929 ts_manager.cc:194] Registered new tserver with Master: e37e97b661f04f24ac14a7101ebaf400 (127.3.211.193:44327)
I20260925 15:24:15.908555 20929 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.193@KRBTEST.COM'} at 127.3.211.193:57907
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab.
Entry for principal kudu/127.3.211.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab.
I20260925 15:24:15.929291 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.194:0
--local_ip_for_outbound_sockets=127.3.211.194
--webserver_interface=127.3.211.194
--webserver_port=0
--tserver_master_addrs=127.3.211.254:43173
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab
--principal=kudu/127.3.211.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:36559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:24:16.015347 21125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:16.015585 21125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:16.015614 21125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:16.016897 21125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:16.016956 21125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:16.016985 21125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.194
I20260925 15:24:16.018540 21125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:36559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab
--principal=kudu/127.3.211.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.211.194
--webserver_port=0
--tserver_master_addrs=127.3.211.254:43173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.194
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:24:16.018911 21125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:16.019197 21125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:16.024497 21131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:16.024652 21125 server_base.cc:1034] running on GCE node
W20260925 15:24:16.024487 21130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:16.024492 21133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:16.024937 21125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:16.025205 21125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:16.026392 21125 hybrid_clock.cc:648] HybridClock initialized: now 1790349856026387 us; error 40 us; skew 500 ppm
Sep 25 15:24:16 dist-test-slave-4bk3 krb5kdc[20884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349856, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:24:16.029089 21125 init.cc:377] Logged in from keytab as kudu/127.3.211.194@KRBTEST.COM (short username kudu)
I20260925 15:24:16.029611 21125 webserver.cc:460] Webserver started at http://127.3.211.194:45367/ using document root <none> and password file <none>
I20260925 15:24:16.029865 21125 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:16.029922 21125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:16.030063 21125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:16.031227 21125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/ts-1/data/instance:
uuid: "08c17344e25b42e2b7b5b7ed216add85"
format_stamp: "Formatted at 2026-09-25 15:24:16 on dist-test-slave-4bk3"
server_key: "4e07e2f0d32558ca77a7c84f5ce54708"
server_key_iv: "a29d6faca3aa819d9bc8ee74e21b1fbb"
server_key_version: "encryptionkey@0"
I20260925 15:24:16.031617 21125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/ts-1/wal/instance:
uuid: "08c17344e25b42e2b7b5b7ed216add85"
format_stamp: "Formatted at 2026-09-25 15:24:16 on dist-test-slave-4bk3"
server_key: "4e07e2f0d32558ca77a7c84f5ce54708"
server_key_iv: "a29d6faca3aa819d9bc8ee74e21b1fbb"
server_key_version: "encryptionkey@0"
I20260925 15:24:16.033203 21125 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:24:16.034243 21140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:16.034701 21125 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:24:16.034817 21125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/ts-1/wal
uuid: "08c17344e25b42e2b7b5b7ed216add85"
format_stamp: "Formatted at 2026-09-25 15:24:16 on dist-test-slave-4bk3"
server_key: "4e07e2f0d32558ca77a7c84f5ce54708"
server_key_iv: "a29d6faca3aa819d9bc8ee74e21b1fbb"
server_key_version: "encryptionkey@0"
I20260925 15:24:16.034945 21125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:16.055078 21125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:16.058089 21125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:16.058228 21125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:16.058542 21125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:24:16.058866 21125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:24:16.058921 21125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:16.058956 21125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:24:16.058986 21125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:16.065618 21125 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.194:43849
I20260925 15:24:16.065728 21253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.194:43849 every 8 connection(s)
I20260925 15:24:16.066047 21125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
Sep 25 15:24:16 dist-test-slave-4bk3 krb5kdc[20884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349856, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.194@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:24:16.075500 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 21125
I20260925 15:24:16.075611 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/ts-1/wal/instance
I20260925 15:24:16.075804 3919 external_mini_cluster.cc:1428] Setting key 642dc8daf90f72e05d8de26576cf6d22
I20260925 15:24:16.076792 20983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:16.066996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:52757 (local address 127.3.211.254:43173)
0925 15:24:16.067090 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:16.067092 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:16.067331 (+ 239us) server_negotiation.cc:407] Connection header received
0925 15:24:16.067821 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:16.067822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:16.067834 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:16.067902 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:16.068979 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:16.069501 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:16.070570 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:16.070777 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:16.070847 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:16.073658 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:16.073660 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:16.073660 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:16.073668 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:16.075060 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:16.075507 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:16.075508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:16.075508 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:16.075581 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:16.075979 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:16.075981 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:16.075981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:16.076144 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:16.076213 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:16.076596 (+ 383us) server_negotiation.cc:299] Negotiation successful
0925 15:24:16.076699 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 15:24:16.077028 21256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:16.066961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:43173 (local address 127.3.211.194:52757)
0925 15:24:16.067174 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:16.067219 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:16.067714 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:16.067972 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:16.067975 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:16.068638 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:24:16.068851 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:16.068856 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:16.069605 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:16.069606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:16.070459 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:16.070461 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:16.070885 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:16.070886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:16.071010 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:16.071728 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:24:16.071737 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:24:16.073548 (+ 1811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:24:16.075162 (+ 1614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:16.075170 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:16.075172 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:16.075425 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:16.075731 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:16.075732 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:16.075732 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:16.075882 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:16.076204 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:24:16.076211 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:24:16.076436 (+ 225us) client_negotiation.cc:769] Sending connection context
0925 15:24:16.076576 (+ 140us) client_negotiation.cc:240] Negotiation successful
0925 15:24:16.076762 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":132,"threads_started":1}
I20260925 15:24:16.077510 21254 heartbeater.cc:347] Connected to a master server at 127.3.211.254:43173
I20260925 15:24:16.077632 21254 heartbeater.cc:464] Registering TS with master...
I20260925 15:24:16.077847 21254 heartbeater.cc:511] Master 127.3.211.254:43173 requested a full tablet report, sending...
I20260925 15:24:16.078230 20929 ts_manager.cc:194] Registered new tserver with Master: 08c17344e25b42e2b7b5b7ed216add85 (127.3.211.194:43849)
I20260925 15:24:16.078576 20929 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.194@KRBTEST.COM'} at 127.3.211.194:52757
WARNING: no policy specified for kudu/127.3.211.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab.
Entry for principal kudu/127.3.211.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab.
I20260925 15:24:16.104212 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.195:0
--local_ip_for_outbound_sockets=127.3.211.195
--webserver_interface=127.3.211.195
--webserver_port=0
--tserver_master_addrs=127.3.211.254:43173
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab
--principal=kudu/127.3.211.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:36559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:24:16.192020 21259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:16.192304 21259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:16.192376 21259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:16.194044 21259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:16.194154 21259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:16.194211 21259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.195
I20260925 15:24:16.195971 21259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:36559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab
--principal=kudu/127.3.211.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.211.195
--webserver_port=0
--tserver_master_addrs=127.3.211.254:43173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.195
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:24:16.196370 21259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:16.196694 21259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:16.203495 21265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:16.203495 21264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:16.203517 21267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:16.204607 21259 server_base.cc:1034] running on GCE node
I20260925 15:24:16.204849 21259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:16.205103 21259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:16.206357 21259 hybrid_clock.cc:648] HybridClock initialized: now 1790349856206336 us; error 30 us; skew 500 ppm
Sep 25 15:24:16 dist-test-slave-4bk3 krb5kdc[20884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349856, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:24:16.209378 21259 init.cc:377] Logged in from keytab as kudu/127.3.211.195@KRBTEST.COM (short username kudu)
I20260925 15:24:16.209914 21259 webserver.cc:460] Webserver started at http://127.3.211.195:43869/ using document root <none> and password file <none>
I20260925 15:24:16.210171 21259 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:16.210242 21259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:16.210410 21259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:16.211544 21259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/ts-2/data/instance:
uuid: "e26c4011b40640b2918619da6230bd23"
format_stamp: "Formatted at 2026-09-25 15:24:16 on dist-test-slave-4bk3"
server_key: "2e9ec49ea75a649ce815ab8cbd821016"
server_key_iv: "397a97195ee5ae162f28bcdd5eb1b854"
server_key_version: "encryptionkey@0"
I20260925 15:24:16.211925 21259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/ts-2/wal/instance:
uuid: "e26c4011b40640b2918619da6230bd23"
format_stamp: "Formatted at 2026-09-25 15:24:16 on dist-test-slave-4bk3"
server_key: "2e9ec49ea75a649ce815ab8cbd821016"
server_key_iv: "397a97195ee5ae162f28bcdd5eb1b854"
server_key_version: "encryptionkey@0"
I20260925 15:24:16.213553 21259 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:24:16.214454 21274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:16.214690 21259 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:24:16.214802 21259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/ts-2/wal
uuid: "e26c4011b40640b2918619da6230bd23"
format_stamp: "Formatted at 2026-09-25 15:24:16 on dist-test-slave-4bk3"
server_key: "2e9ec49ea75a649ce815ab8cbd821016"
server_key_iv: "397a97195ee5ae162f28bcdd5eb1b854"
server_key_version: "encryptionkey@0"
I20260925 15:24:16.214885 21259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:16.228899 21259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:16.231827 21259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:16.231986 21259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:16.232228 21259 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:24:16.232702 21259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:24:16.232786 21259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:16.232838 21259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:24:16.232869 21259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:16.241547 21259 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.195:33601
I20260925 15:24:16.241643 21387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.195:33601 every 8 connection(s)
I20260925 15:24:16.242092 21259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
Sep 25 15:24:16 dist-test-slave-4bk3 krb5kdc[20884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349856, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.195@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:24:16.250231 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 21259
I20260925 15:24:16.250312 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/ts-2/wal/instance
I20260925 15:24:16.250504 3919 external_mini_cluster.cc:1428] Setting key 04b4eeb48d704eb6c23f81a697a83a3c
I20260925 15:24:16.253973 20983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:16.243125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:42195 (local address 127.3.211.254:43173)
0925 15:24:16.243203 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:16.243206 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:16.243368 (+ 162us) server_negotiation.cc:407] Connection header received
0925 15:24:16.243941 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:16.243942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:16.243959 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:16.244080 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:16.245321 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:16.245913 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:16.247277 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:16.247627 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:16.247713 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:16.250442 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:16.250445 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:16.250445 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:16.250457 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:16.252416 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:16.252913 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:16.252914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:16.252915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:16.252976 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:16.253255 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:16.253256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:16.253256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:16.253400 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:16.253477 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:16.253780 (+ 303us) server_negotiation.cc:299] Negotiation successful
0925 15:24:16.253871 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 15:24:16.254194 21390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:16.242972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:43173 (local address 127.3.211.195:42195)
0925 15:24:16.243155 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:16.243206 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:16.243759 (+ 553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:16.244197 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:16.244200 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:16.244943 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:24:16.245160 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:16.245166 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:16.246090 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:16.246091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:16.246989 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:16.246991 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:16.247757 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:16.247758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:16.247898 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:16.248654 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:24:16.248664 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:24:16.250249 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:24:16.252538 (+ 2289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:16.252549 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:16.252552 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:16.252817 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:16.253057 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:16.253058 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:16.253058 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:16.253184 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:16.253480 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:24:16.253486 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:24:16.253642 (+ 156us) client_negotiation.cc:769] Sending connection context
0925 15:24:16.253757 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 15:24:16.253943 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":102,"threads_started":1}
I20260925 15:24:16.254902 21388 heartbeater.cc:347] Connected to a master server at 127.3.211.254:43173
I20260925 15:24:16.254998 21388 heartbeater.cc:464] Registering TS with master...
I20260925 15:24:16.255172 21388 heartbeater.cc:511] Master 127.3.211.254:43173 requested a full tablet report, sending...
I20260925 15:24:16.255594 20929 ts_manager.cc:194] Registered new tserver with Master: e26c4011b40640b2918619da6230bd23 (127.3.211.195:33601)
I20260925 15:24:16.256007 20929 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.195@KRBTEST.COM'} at 127.3.211.195:42195
I20260925 15:24:16.262671 3919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:24:16.271582 20983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:16.264144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37986 (local address 127.3.211.254:43173)
0925 15:24:16.264234 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:16.264236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:16.264356 (+ 120us) server_negotiation.cc:407] Connection header received
0925 15:24:16.264490 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:16.264491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:16.264503 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:16.264555 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:16.265136 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:16.265646 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:16.266495 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:16.266709 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:16.266798 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:16.268471 (+ 1673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:16.268473 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:16.268474 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:16.268482 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:16.270048 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:16.270535 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:16.270537 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:16.270537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:16.270606 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:16.270843 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:16.270844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:16.270844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:16.271063 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:16.271187 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:16.271359 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 15:24:16.271464 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 15:24:16.273713 20929 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37986:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:24:16.275516 20929 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:24:16.283286 21404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:16.278624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38022 (local address 127.3.211.193:44327)
0925 15:24:16.279043 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:16.279106 (+ 63us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:16.279121 (+ 15us) server_negotiation.cc:407] Connection header received
0925 15:24:16.279173 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:16.279176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:16.279196 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:16.279348 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:24:16.279617 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:16.280876 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:16.282211 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:16.282795 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:16.282903 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:16.282968 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:16.283135 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 15:24:16.283196 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":91,"threads_started":1}
I20260925 15:24:16.283340 21401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:16.278351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:44327 (local address 127.0.0.1:38022)
0925 15:24:16.278742 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:16.278751 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:16.278788 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:16.279419 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:16.279420 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:16.279424 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:24:16.279492 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:16.279494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:16.280977 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:16.280979 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:16.282078 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:16.282081 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:16.282904 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:16.282905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:16.282997 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:16.283012 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 15:24:16.283106 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 15:24:16.283169 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":106,"threads_started":1}
I20260925 15:24:16.283344 21400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:16.278284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.195:33601 (local address 127.0.0.1:58472)
0925 15:24:16.278522 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:16.278562 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:16.278634 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:16.279143 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:16.279149 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:16.279153 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:24:16.279293 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:16.279296 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:16.280728 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:16.280730 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:16.282065 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:16.282069 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:16.283017 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:16.283018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:16.283087 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:16.283091 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:24:16.283147 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 15:24:16.283210 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":11,"thread_start_us":73,"threads_started":1}
I20260925 15:24:16.283535 21402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:16.278365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58472 (local address 127.3.211.195:33601)
0925 15:24:16.278897 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:16.278901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:16.278909 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:24:16.278941 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:16.278942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:16.278954 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:16.279017 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:24:16.279416 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:16.280608 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:16.282144 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:16.282831 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:16.282908 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:16.282978 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:16.283388 (+ 410us) server_negotiation.cc:299] Negotiation successful
0925 15:24:16.283439 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":85,"threads_started":1}
I20260925 15:24:16.283730 21399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:16.278347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.194:43849 (local address 127.0.0.1:47654)
0925 15:24:16.278607 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:16.278624 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:16.278681 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:16.279835 (+ 1154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:16.279836 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:16.279839 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:24:16.279902 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:16.279904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:16.281379 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:16.281381 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:16.282427 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:16.282430 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:16.283499 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:16.283500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:16.283571 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:16.283575 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:24:16.283632 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 15:24:16.283673 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":120,"threads_started":1}
I20260925 15:24:16.283993 21403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:16.278623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47654 (local address 127.3.211.194:43849)
0925 15:24:16.279529 (+ 906us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:16.279532 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:16.279587 (+ 55us) server_negotiation.cc:407] Connection header received
0925 15:24:16.279635 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:16.279637 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:16.279655 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:16.279753 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:24:16.279989 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:16.281245 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:16.282529 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:16.283313 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:16.283387 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:16.283459 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:16.283854 (+ 395us) server_negotiation.cc:299] Negotiation successful
0925 15:24:16.283908 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":865,"thread_start_us":85,"threads_started":1}
I20260925 15:24:16.284420 21322 tablet_service.cc:1467] Processing CreateTablet for tablet e78547409bd645559da0ebc6d4e7f4a7 (DEFAULT_TABLE table=test-table [id=cf7693a8e2b84ee8aa64a39afb769937]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:24:16.284804 21322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e78547409bd645559da0ebc6d4e7f4a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:16.284727 21188 tablet_service.cc:1467] Processing CreateTablet for tablet e78547409bd645559da0ebc6d4e7f4a7 (DEFAULT_TABLE table=test-table [id=cf7693a8e2b84ee8aa64a39afb769937]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:24:16.284996 21188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e78547409bd645559da0ebc6d4e7f4a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:16.284899 21054 tablet_service.cc:1467] Processing CreateTablet for tablet e78547409bd645559da0ebc6d4e7f4a7 (DEFAULT_TABLE table=test-table [id=cf7693a8e2b84ee8aa64a39afb769937]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:24:16.285194 21054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e78547409bd645559da0ebc6d4e7f4a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:16.287981 21405 tablet_bootstrap.cc:492] T e78547409bd645559da0ebc6d4e7f4a7 P 08c17344e25b42e2b7b5b7ed216add85: Bootstrap starting.
I20260925 15:24:16.288542 21407 tablet_bootstrap.cc:492] T e78547409bd645559da0ebc6d4e7f4a7 P e37e97b661f04f24ac14a7101ebaf400: Bootstrap starting.
I20260925 15:24:16.288861 21405 tablet_bootstrap.cc:654] T e78547409bd645559da0ebc6d4e7f4a7 P 08c17344e25b42e2b7b5b7ed216add85: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:16.289124 21407 tablet_bootstrap.cc:654] T e78547409bd645559da0ebc6d4e7f4a7 P e37e97b661f04f24ac14a7101ebaf400: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:16.289480 21405 log.cc:824] T e78547409bd645559da0ebc6d4e7f4a7 P 08c17344e25b42e2b7b5b7ed216add85: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:16.289600 21407 log.cc:824] T e78547409bd645559da0ebc6d4e7f4a7 P e37e97b661f04f24ac14a7101ebaf400: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:16.289966 21406 tablet_bootstrap.cc:492] T e78547409bd645559da0ebc6d4e7f4a7 P e26c4011b40640b2918619da6230bd23: Bootstrap starting.
I20260925 15:24:16.290378 21405 tablet_bootstrap.cc:492] T e78547409bd645559da0ebc6d4e7f4a7 P 08c17344e25b42e2b7b5b7ed216add85: No bootstrap required, opened a new log
I20260925 15:24:16.290395 21407 tablet_bootstrap.cc:492] T e78547409bd645559da0ebc6d4e7f4a7 P e37e97b661f04f24ac14a7101ebaf400: No bootstrap required, opened a new log
I20260925 15:24:16.290458 21405 ts_tablet_manager.cc:1397] T e78547409bd645559da0ebc6d4e7f4a7 P 08c17344e25b42e2b7b5b7ed216add85: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 15:24:16.290457 21407 ts_tablet_manager.cc:1397] T e78547409bd645559da0ebc6d4e7f4a7 P e37e97b661f04f24ac14a7101ebaf400: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:24:16.290769 21406 tablet_bootstrap.cc:654] T e78547409bd645559da0ebc6d4e7f4a7 P e26c4011b40640b2918619da6230bd23: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:16.291478 21406 log.cc:824] T e78547409bd645559da0ebc6d4e7f4a7 P e26c4011b40640b2918619da6230bd23: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:16.292277 21406 tablet_bootstrap.cc:492] T e78547409bd645559da0ebc6d4e7f4a7 P e26c4011b40640b2918619da6230bd23: No bootstrap required, opened a new log
I20260925 15:24:16.292411 21406 ts_tablet_manager.cc:1397] T e78547409bd645559da0ebc6d4e7f4a7 P e26c4011b40640b2918619da6230bd23: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 15:24:16.292579 21405 raft_consensus.cc:348] T e78547409bd645559da0ebc6d4e7f4a7 P 08c17344e25b42e2b7b5b7ed216add85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26c4011b40640b2918619da6230bd23" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 33601 } } peers { permanent_uuid: "e37e97b661f04f24ac14a7101ebaf400" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 44327 } } peers { permanent_uuid: "08c17344e25b42e2b7b5b7ed216add85" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 43849 } }
I20260925 15:24:16.292579 21407 raft_consensus.cc:348] T e78547409bd645559da0ebc6d4e7f4a7 P e37e97b661f04f24ac14a7101ebaf400 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26c4011b40640b2918619da6230bd23" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 33601 } } peers { permanent_uuid: "e37e97b661f04f24ac14a7101ebaf400" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 44327 } } peers { permanent_uuid: "08c17344e25b42e2b7b5b7ed216add85" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 43849 } }
I20260925 15:24:16.292773 21407 raft_consensus.cc:374] T e78547409bd645559da0ebc6d4e7f4a7 P e37e97b661f04f24ac14a7101ebaf400 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:24:16.292773 21405 raft_consensus.cc:374] T e78547409bd645559da0ebc6d4e7f4a7 P 08c17344e25b42e2b7b5b7ed216add85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:24:16.292811 21407 raft_consensus.cc:729] T e78547409bd645559da0ebc6d4e7f4a7 P e37e97b661f04f24ac14a7101ebaf400 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e37e97b661f04f24ac14a7101ebaf400, State: Initialized, Role: FOLLOWER
I20260925 15:24:16.292811 21405 raft_consensus.cc:729] T e78547409bd645559da0ebc6d4e7f4a7 P 08c17344e25b42e2b7b5b7ed216add85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08c17344e25b42e2b7b5b7ed216add85, State: Initialized, Role: FOLLOWER
I20260925 15:24:16.292968 21405 consensus_queue.cc:260] T e78547409bd645559da0ebc6d4e7f4a7 P 08c17344e25b42e2b7b5b7ed216add85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26c4011b40640b2918619da6230bd23" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 33601 } } peers { permanent_uuid: "e37e97b661f04f24ac14a7101ebaf400" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 44327 } } peers { permanent_uuid: "08c17344e25b42e2b7b5b7ed216add85" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 43849 } }
I20260925 15:24:16.292968 21407 consensus_queue.cc:260] T e78547409bd645559da0ebc6d4e7f4a7 P e37e97b661f04f24ac14a7101ebaf400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26c4011b40640b2918619da6230bd23" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 33601 } } peers { permanent_uuid: "e37e97b661f04f24ac14a7101ebaf400" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 44327 } } peers { permanent_uuid: "08c17344e25b42e2b7b5b7ed216add85" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 43849 } }
I20260925 15:24:16.293267 21407 ts_tablet_manager.cc:1428] T e78547409bd645559da0ebc6d4e7f4a7 P e37e97b661f04f24ac14a7101ebaf400: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:24:16.293267 21405 ts_tablet_manager.cc:1428] T e78547409bd645559da0ebc6d4e7f4a7 P 08c17344e25b42e2b7b5b7ed216add85: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260925 15:24:16.293421 21120 heartbeater.cc:503] Master 127.3.211.254:43173 was elected leader, sending a full tablet report...
I20260925 15:24:16.293443 21254 heartbeater.cc:503] Master 127.3.211.254:43173 was elected leader, sending a full tablet report...
I20260925 15:24:16.295413 21406 raft_consensus.cc:348] T e78547409bd645559da0ebc6d4e7f4a7 P e26c4011b40640b2918619da6230bd23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26c4011b40640b2918619da6230bd23" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 33601 } } peers { permanent_uuid: "e37e97b661f04f24ac14a7101ebaf400" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 44327 } } peers { permanent_uuid: "08c17344e25b42e2b7b5b7ed216add85" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 43849 } }
I20260925 15:24:16.295614 21406 raft_consensus.cc:374] T e78547409bd645559da0ebc6d4e7f4a7 P e26c4011b40640b2918619da6230bd23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:24:16.295644 21406 raft_consensus.cc:729] T e78547409bd645559da0ebc6d4e7f4a7 P e26c4011b40640b2918619da6230bd23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e26c4011b40640b2918619da6230bd23, State: Initialized, Role: FOLLOWER
I20260925 15:24:16.295732 21406 consensus_queue.cc:260] T e78547409bd645559da0ebc6d4e7f4a7 P e26c4011b40640b2918619da6230bd23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26c4011b40640b2918619da6230bd23" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 33601 } } peers { permanent_uuid: "e37e97b661f04f24ac14a7101ebaf400" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 44327 } } peers { permanent_uuid: "08c17344e25b42e2b7b5b7ed216add85" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 43849 } }
I20260925 15:24:16.296002 21406 ts_tablet_manager.cc:1428] T e78547409bd645559da0ebc6d4e7f4a7 P e26c4011b40640b2918619da6230bd23: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260925 15:24:16.296012 21388 heartbeater.cc:503] Master 127.3.211.254:43173 was elected leader, sending a full tablet report...
W20260925 15:24:16.316828 21255 tablet.cc:2369] T e78547409bd645559da0ebc6d4e7f4a7 P 08c17344e25b42e2b7b5b7ed216add85: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:24:16.394848 21121 tablet.cc:2369] T e78547409bd645559da0ebc6d4e7f4a7 P e37e97b661f04f24ac14a7101ebaf400: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:24:16.430518 21412 raft_consensus.cc:482] T e78547409bd645559da0ebc6d4e7f4a7 P 08c17344e25b42e2b7b5b7ed216add85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:24:16.430653 21412 raft_consensus.cc:504] T e78547409bd645559da0ebc6d4e7f4a7 P 08c17344e25b42e2b7b5b7ed216add85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26c4011b40640b2918619da6230bd23" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 33601 } } peers { permanent_uuid: "e37e97b661f04f24ac14a7101ebaf400" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 44327 } } peers { permanent_uuid: "08c17344e25b42e2b7b5b7ed216add85" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 43849 } }
I20260925 15:24:16.431125 21412 leader_election.cc:288] T e78547409bd645559da0ebc6d4e7f4a7 P 08c17344e25b42e2b7b5b7ed216add85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e26c4011b40640b2918619da6230bd23 (127.3.211.195:33601), e37e97b661f04f24ac14a7101ebaf400 (127.3.211.193:44327)
I20260925 15:24:16.435377 21404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:16.431472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:51129 (local address 127.3.211.193:44327)
0925 15:24:16.431525 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:16.431527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:16.431643 (+ 116us) server_negotiation.cc:407] Connection header received
0925 15:24:16.431679 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:16.431680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:16.431688 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:16.431754 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:24:16.431885 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:16.432603 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:16.434261 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:16.434859 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:16.434959 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:16.434969 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:16.435244 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 15:24:16.435292 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 15:24:16.435364 21402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:16.431396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:54681 (local address 127.3.211.195:33601)
0925 15:24:16.431463 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:16.431465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:16.431476 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:24:16.431506 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:16.431507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:16.431517 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:16.431590 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:24:16.431752 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:16.432581 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:16.434261 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:16.434878 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:16.434959 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:16.434969 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:16.435244 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 15:24:16.435292 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 15:24:16.435415 21256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:16.431321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.195:33601 (local address 127.3.211.194:54681)
0925 15:24:16.431366 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:16.431378 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:16.431430 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:16.431597 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:16.431598 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:16.431601 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:24:16.431679 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:16.431681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:16.432708 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:16.432709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:16.434125 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:16.434129 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:16.435007 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:16.435008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:16.435128 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:16.435131 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:24:16.435211 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 15:24:16.435275 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22,"mutex_wait_us":138}
I20260925 15:24:16.435425 21414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:16.431400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:44327 (local address 127.3.211.194:51129)
0925 15:24:16.431555 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:16.431565 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:16.431619 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:16.431759 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:16.431760 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:16.431761 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:24:16.431817 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:16.431818 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:16.432708 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:16.432709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:16.434125 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:16.434129 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:16.435007 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:16.435008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:16.435126 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:16.435132 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 15:24:16.435211 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 15:24:16.435275 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":91,"thread_start_us":105,"threads_started":1}
I20260925 15:24:16.435825 21342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78547409bd645559da0ebc6d4e7f4a7" candidate_uuid: "08c17344e25b42e2b7b5b7ed216add85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e26c4011b40640b2918619da6230bd23" is_pre_election: true
I20260925 15:24:16.435897 21074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78547409bd645559da0ebc6d4e7f4a7" candidate_uuid: "08c17344e25b42e2b7b5b7ed216add85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e37e97b661f04f24ac14a7101ebaf400" is_pre_election: true
I20260925 15:24:16.436029 21342 raft_consensus.cc:2454] T e78547409bd645559da0ebc6d4e7f4a7 P e26c4011b40640b2918619da6230bd23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08c17344e25b42e2b7b5b7ed216add85 in term 0.
I20260925 15:24:16.436054 21074 raft_consensus.cc:2454] T e78547409bd645559da0ebc6d4e7f4a7 P e37e97b661f04f24ac14a7101ebaf400 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08c17344e25b42e2b7b5b7ed216add85 in term 0.
I20260925 15:24:16.436302 21143 leader_election.cc:302] T e78547409bd645559da0ebc6d4e7f4a7 P 08c17344e25b42e2b7b5b7ed216add85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08c17344e25b42e2b7b5b7ed216add85, e37e97b661f04f24ac14a7101ebaf400; no voters:
I20260925 15:24:16.436478 21412 raft_consensus.cc:2781] T e78547409bd645559da0ebc6d4e7f4a7 P 08c17344e25b42e2b7b5b7ed216add85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:24:16.436542 21412 raft_consensus.cc:482] T e78547409bd645559da0ebc6d4e7f4a7 P 08c17344e25b42e2b7b5b7ed216add85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:24:16.436558 21412 raft_consensus.cc:3037] T e78547409bd645559da0ebc6d4e7f4a7 P 08c17344e25b42e2b7b5b7ed216add85 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:16.437290 21412 raft_consensus.cc:504] T e78547409bd645559da0ebc6d4e7f4a7 P 08c17344e25b42e2b7b5b7ed216add85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26c4011b40640b2918619da6230bd23" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 33601 } } peers { permanent_uuid: "e37e97b661f04f24ac14a7101ebaf400" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 44327 } } peers { permanent_uuid: "08c17344e25b42e2b7b5b7ed216add85" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 43849 } }
I20260925 15:24:16.437475 21412 leader_election.cc:288] T e78547409bd645559da0ebc6d4e7f4a7 P 08c17344e25b42e2b7b5b7ed216add85 [CANDIDATE]: Term 1 election: Requested vote from peers e26c4011b40640b2918619da6230bd23 (127.3.211.195:33601), e37e97b661f04f24ac14a7101ebaf400 (127.3.211.193:44327)
I20260925 15:24:16.437683 21342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78547409bd645559da0ebc6d4e7f4a7" candidate_uuid: "08c17344e25b42e2b7b5b7ed216add85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e26c4011b40640b2918619da6230bd23"
I20260925 15:24:16.437732 21074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e78547409bd645559da0ebc6d4e7f4a7" candidate_uuid: "08c17344e25b42e2b7b5b7ed216add85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e37e97b661f04f24ac14a7101ebaf400"
I20260925 15:24:16.437781 21342 raft_consensus.cc:3037] T e78547409bd645559da0ebc6d4e7f4a7 P e26c4011b40640b2918619da6230bd23 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:16.437850 21074 raft_consensus.cc:3037] T e78547409bd645559da0ebc6d4e7f4a7 P e37e97b661f04f24ac14a7101ebaf400 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:16.438575 21342 raft_consensus.cc:2454] T e78547409bd645559da0ebc6d4e7f4a7 P e26c4011b40640b2918619da6230bd23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08c17344e25b42e2b7b5b7ed216add85 in term 1.
I20260925 15:24:16.438661 21074 raft_consensus.cc:2454] T e78547409bd645559da0ebc6d4e7f4a7 P e37e97b661f04f24ac14a7101ebaf400 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08c17344e25b42e2b7b5b7ed216add85 in term 1.
I20260925 15:24:16.438822 21143 leader_election.cc:302] T e78547409bd645559da0ebc6d4e7f4a7 P 08c17344e25b42e2b7b5b7ed216add85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08c17344e25b42e2b7b5b7ed216add85, e37e97b661f04f24ac14a7101ebaf400; no voters:
I20260925 15:24:16.438978 21412 raft_consensus.cc:2781] T e78547409bd645559da0ebc6d4e7f4a7 P 08c17344e25b42e2b7b5b7ed216add85 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:24:16.439126 21412 raft_consensus.cc:686] T e78547409bd645559da0ebc6d4e7f4a7 P 08c17344e25b42e2b7b5b7ed216add85 [term 1 LEADER]: Becoming Leader. State: Replica: 08c17344e25b42e2b7b5b7ed216add85, State: Running, Role: LEADER
I20260925 15:24:16.439231 21412 consensus_queue.cc:237] T e78547409bd645559da0ebc6d4e7f4a7 P 08c17344e25b42e2b7b5b7ed216add85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26c4011b40640b2918619da6230bd23" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 33601 } } peers { permanent_uuid: "e37e97b661f04f24ac14a7101ebaf400" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 44327 } } peers { permanent_uuid: "08c17344e25b42e2b7b5b7ed216add85" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 43849 } }
I20260925 15:24:16.440183 20929 catalog_manager.cc:5495] T e78547409bd645559da0ebc6d4e7f4a7 P 08c17344e25b42e2b7b5b7ed216add85 reported cstate change: term changed from 0 to 1, leader changed from <none> to 08c17344e25b42e2b7b5b7ed216add85 (127.3.211.194). New cstate: current_term: 1 leader_uuid: "08c17344e25b42e2b7b5b7ed216add85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e26c4011b40640b2918619da6230bd23" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 33601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e37e97b661f04f24ac14a7101ebaf400" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 44327 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08c17344e25b42e2b7b5b7ed216add85" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 43849 } health_report { overall_health: HEALTHY } } }
I20260925 15:24:16.469450 21403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:16.466637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47666 (local address 127.3.211.194:43849)
0925 15:24:16.466724 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:16.466726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:16.466736 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:24:16.466785 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:16.466786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:16.466805 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:16.466895 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:24:16.467056 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:16.467609 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:16.468526 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:16.468742 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:16.468830 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:16.469012 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:24:16.469144 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:24:16.469247 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:16.469294 (+ 47us) server_negotiation.cc:299] Negotiation successful
0925 15:24:16.469318 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 15:24:16.471463 21074 raft_consensus.cc:1261] T e78547409bd645559da0ebc6d4e7f4a7 P e37e97b661f04f24ac14a7101ebaf400 [term 1 FOLLOWER]: Refusing update from remote peer 08c17344e25b42e2b7b5b7ed216add85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:24:16.471457 21342 raft_consensus.cc:1261] T e78547409bd645559da0ebc6d4e7f4a7 P e26c4011b40640b2918619da6230bd23 [term 1 FOLLOWER]: Refusing update from remote peer 08c17344e25b42e2b7b5b7ed216add85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:24:16.471920 21415 consensus_queue.cc:1035] T e78547409bd645559da0ebc6d4e7f4a7 P 08c17344e25b42e2b7b5b7ed216add85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e37e97b661f04f24ac14a7101ebaf400" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 44327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:24:16.472087 21412 consensus_queue.cc:1035] T e78547409bd645559da0ebc6d4e7f4a7 P 08c17344e25b42e2b7b5b7ed216add85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e26c4011b40640b2918619da6230bd23" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 33601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:24:16.474661 21418 mvcc.cc:205] Tried to move back new op lower bound from 7333273012104077312 to 7333273011975749632. Current Snapshot: MvccSnapshot[applied={T|T < 7333273012104077312}]
I20260925 15:24:16.475268 21420 mvcc.cc:205] Tried to move back new op lower bound from 7333273012104077312 to 7333273011975749632. Current Snapshot: MvccSnapshot[applied={T|T < 7333273012104077312}]
I20260925 15:24:16.477535 20928 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:24:16.477640 20928 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37986:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:24:16.478404 20928 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 813298235fc34d168e4f56b77dd5361c: Sending DeleteTablet for 3 replicas of tablet e78547409bd645559da0ebc6d4e7f4a7
I20260925 15:24:16.478739 21188 tablet_service.cc:1514] Processing DeleteTablet for tablet e78547409bd645559da0ebc6d4e7f4a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:24:16 UTC) from {username='kudu', principal='kudu/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:47654
I20260925 15:24:16.478787 21322 tablet_service.cc:1514] Processing DeleteTablet for tablet e78547409bd645559da0ebc6d4e7f4a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:24:16 UTC) from {username='kudu', principal='kudu/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:58472
I20260925 15:24:16.479032 21428 tablet_replica.cc:330] T e78547409bd645559da0ebc6d4e7f4a7 P 08c17344e25b42e2b7b5b7ed216add85: stopping tablet replica
I20260925 15:24:16.479072 21054 tablet_service.cc:1514] Processing DeleteTablet for tablet e78547409bd645559da0ebc6d4e7f4a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:24:16 UTC) from {username='kudu', principal='kudu/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:38022
I20260925 15:24:16.479182 21428 raft_consensus.cc:2229] T e78547409bd645559da0ebc6d4e7f4a7 P 08c17344e25b42e2b7b5b7ed216add85 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:24:16.479276 21428 raft_consensus.cc:2258] T e78547409bd645559da0ebc6d4e7f4a7 P 08c17344e25b42e2b7b5b7ed216add85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:24:16.479358 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 20991
I20260925 15:24:16.479857 21429 tablet_replica.cc:330] T e78547409bd645559da0ebc6d4e7f4a7 P e26c4011b40640b2918619da6230bd23: stopping tablet replica
I20260925 15:24:16.479977 21429 raft_consensus.cc:2229] T e78547409bd645559da0ebc6d4e7f4a7 P e26c4011b40640b2918619da6230bd23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:24:16.480168 21429 raft_consensus.cc:2258] T e78547409bd645559da0ebc6d4e7f4a7 P e26c4011b40640b2918619da6230bd23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:24:16.480372 21428 ts_tablet_manager.cc:1905] T e78547409bd645559da0ebc6d4e7f4a7 P 08c17344e25b42e2b7b5b7ed216add85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:24:16.480736 21429 ts_tablet_manager.cc:1905] T e78547409bd645559da0ebc6d4e7f4a7 P e26c4011b40640b2918619da6230bd23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:24:16.482405 21429 ts_tablet_manager.cc:1918] T e78547409bd645559da0ebc6d4e7f4a7 P e26c4011b40640b2918619da6230bd23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:24:16.482517 21428 ts_tablet_manager.cc:1918] T e78547409bd645559da0ebc6d4e7f4a7 P 08c17344e25b42e2b7b5b7ed216add85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:24:16.482580 21429 log.cc:1196] T e78547409bd645559da0ebc6d4e7f4a7 P e26c4011b40640b2918619da6230bd23: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/ts-2/wal/wals/e78547409bd645559da0ebc6d4e7f4a7
I20260925 15:24:16.482632 21428 log.cc:1196] T e78547409bd645559da0ebc6d4e7f4a7 P 08c17344e25b42e2b7b5b7ed216add85: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790349610595785-3919-0/minicluster-data/ts-1/wal/wals/e78547409bd645559da0ebc6d4e7f4a7
I20260925 15:24:16.482949 21428 ts_tablet_manager.cc:1939] T e78547409bd645559da0ebc6d4e7f4a7 P 08c17344e25b42e2b7b5b7ed216add85: Deleting consensus metadata
I20260925 15:24:16.483047 21429 ts_tablet_manager.cc:1939] T e78547409bd645559da0ebc6d4e7f4a7 P e26c4011b40640b2918619da6230bd23: Deleting consensus metadata
I20260925 15:24:16.483572 20916 catalog_manager.cc:4841] TS 08c17344e25b42e2b7b5b7ed216add85 (127.3.211.194:43849): tablet e78547409bd645559da0ebc6d4e7f4a7 (table test-table [id=cf7693a8e2b84ee8aa64a39afb769937]) successfully deleted
I20260925 15:24:16.483572 20914 catalog_manager.cc:4841] TS e26c4011b40640b2918619da6230bd23 (127.3.211.195:33601): tablet e78547409bd645559da0ebc6d4e7f4a7 (table test-table [id=cf7693a8e2b84ee8aa64a39afb769937]) successfully deleted
W20260925 15:24:16.487152 20915 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.211.193:44327 (error 108)
I20260925 15:24:16.487267 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 21125
I20260925 15:24:16.489001 21399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:16.488846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:44327 (local address 127.0.0.1:38028)
0925 15:24:16.488911 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:16.488952 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.193:44327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 15:24:16.489156 20915 catalog_manager.cc:4584] TS e37e97b661f04f24ac14a7101ebaf400 (127.3.211.193:44327): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e78547409bd645559da0ebc6d4e7f4a7: Network error: Client connection negotiation failed: client connection to 127.3.211.193:44327: connect: Connection refused (error 111)
I20260925 15:24:16.492095 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 21259
I20260925 15:24:16.496932 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 20897
2026-09-25T15:24:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2163 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T15:24:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:24:16Z Disabled control of system clock
I20260925 15:24:16.526234 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:37305
--webserver_interface=127.3.211.254
--webserver_port=0
--builtin_ntp_servers=127.3.211.212:39503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.211.254:37305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:24:16.617004 21435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:16.617254 21435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:16.617275 21435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:16.619009 21435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:24:16.619088 21435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:16.619104 21435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:16.619119 21435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:24:16.619133 21435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:24:16.621026 21435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:39503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.211.254:37305
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:37305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.211.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:24:16.621368 21435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:16.621660 21435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:16.626876 21443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:16.626916 21440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:16.626916 21441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:16.627336 21435 server_base.cc:1034] running on GCE node
I20260925 15:24:16.627663 21435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:16.628006 21435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:16.629212 21435 hybrid_clock.cc:648] HybridClock initialized: now 1790349856629180 us; error 49 us; skew 500 ppm
I20260925 15:24:16.630625 21435 webserver.cc:460] Webserver started at http://127.3.211.254:39723/ using document root <none> and password file <none>
I20260925 15:24:16.630893 21435 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:16.630970 21435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:16.631150 21435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:16.632378 21435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/master-0/data/instance:
uuid: "e2c3e28e73f84c9e9a440334270c2ef5"
format_stamp: "Formatted at 2026-09-25 15:24:16 on dist-test-slave-4bk3"
server_key: "2f44dafe4edbbd9005739022f6024fdd"
server_key_iv: "7487c7622af31a41b05f4331244d6468"
server_key_version: "encryptionkey@0"
I20260925 15:24:16.632802 21435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/master-0/wal/instance:
uuid: "e2c3e28e73f84c9e9a440334270c2ef5"
format_stamp: "Formatted at 2026-09-25 15:24:16 on dist-test-slave-4bk3"
server_key: "2f44dafe4edbbd9005739022f6024fdd"
server_key_iv: "7487c7622af31a41b05f4331244d6468"
server_key_version: "encryptionkey@0"
I20260925 15:24:16.634291 21435 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:24:16.635771 21449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:16.636114 21435 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:24:16.636260 21435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/master-0/wal
uuid: "e2c3e28e73f84c9e9a440334270c2ef5"
format_stamp: "Formatted at 2026-09-25 15:24:16 on dist-test-slave-4bk3"
server_key: "2f44dafe4edbbd9005739022f6024fdd"
server_key_iv: "7487c7622af31a41b05f4331244d6468"
server_key_version: "encryptionkey@0"
I20260925 15:24:16.636376 21435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:16.669466 21435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:16.669869 21435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:16.670047 21435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:16.674813 21435 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.254:37305
I20260925 15:24:16.675035 21501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.254:37305 every 8 connection(s)
I20260925 15:24:16.675311 21435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
I20260925 15:24:16.676538 21502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:16.678982 21502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2c3e28e73f84c9e9a440334270c2ef5: Bootstrap starting.
I20260925 15:24:16.679643 21502 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2c3e28e73f84c9e9a440334270c2ef5: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:16.679984 21502 log.cc:824] T 00000000000000000000000000000000 P e2c3e28e73f84c9e9a440334270c2ef5: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:16.680956 21502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2c3e28e73f84c9e9a440334270c2ef5: No bootstrap required, opened a new log
I20260925 15:24:16.682760 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 21435
I20260925 15:24:16.682665 21502 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2c3e28e73f84c9e9a440334270c2ef5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2c3e28e73f84c9e9a440334270c2ef5" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 37305 } }
I20260925 15:24:16.683000 21502 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2c3e28e73f84c9e9a440334270c2ef5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:24:16.683029 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/master-0/wal/instance
I20260925 15:24:16.683063 21502 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2c3e28e73f84c9e9a440334270c2ef5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2c3e28e73f84c9e9a440334270c2ef5, State: Initialized, Role: FOLLOWER
I20260925 15:24:16.683250 21502 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2c3e28e73f84c9e9a440334270c2ef5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2c3e28e73f84c9e9a440334270c2ef5" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 37305 } }
I20260925 15:24:16.683343 21502 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2c3e28e73f84c9e9a440334270c2ef5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:24:16.683369 21502 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2c3e28e73f84c9e9a440334270c2ef5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:24:16.683370 3919 external_mini_cluster.cc:1428] Setting key 056ef0d464f197ba2f59ba08dc2865f7
I20260925 15:24:16.683422 21502 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2c3e28e73f84c9e9a440334270c2ef5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:16.684026 21502 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2c3e28e73f84c9e9a440334270c2ef5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2c3e28e73f84c9e9a440334270c2ef5" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 37305 } }
I20260925 15:24:16.684140 21502 leader_election.cc:302] T 00000000000000000000000000000000 P e2c3e28e73f84c9e9a440334270c2ef5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2c3e28e73f84c9e9a440334270c2ef5; no voters:
I20260925 15:24:16.684389 21502 leader_election.cc:288] T 00000000000000000000000000000000 P e2c3e28e73f84c9e9a440334270c2ef5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:24:16.684715 21506 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2c3e28e73f84c9e9a440334270c2ef5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:24:16.684726 21502 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2c3e28e73f84c9e9a440334270c2ef5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:24:16.685644 21506 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2c3e28e73f84c9e9a440334270c2ef5 [term 1 LEADER]: Becoming Leader. State: Replica: e2c3e28e73f84c9e9a440334270c2ef5, State: Running, Role: LEADER
I20260925 15:24:16.685810 21506 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2c3e28e73f84c9e9a440334270c2ef5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2c3e28e73f84c9e9a440334270c2ef5" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 37305 } }
I20260925 15:24:16.686425 21512 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2c3e28e73f84c9e9a440334270c2ef5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2c3e28e73f84c9e9a440334270c2ef5. Latest consensus state: current_term: 1 leader_uuid: "e2c3e28e73f84c9e9a440334270c2ef5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2c3e28e73f84c9e9a440334270c2ef5" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 37305 } } }
I20260925 15:24:16.686511 21512 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2c3e28e73f84c9e9a440334270c2ef5 [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:16.686610 21505 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2c3e28e73f84c9e9a440334270c2ef5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2c3e28e73f84c9e9a440334270c2ef5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2c3e28e73f84c9e9a440334270c2ef5" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 37305 } } }
I20260925 15:24:16.686834 21505 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2c3e28e73f84c9e9a440334270c2ef5 [sys.catalog]: This master's current role is: LEADER
W20260925 15:24:16.687225 21521 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e2c3e28e73f84c9e9a440334270c2ef5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:24:16.687332 21521 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:24:16.687433 21522 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:24:16.687634 21522 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:24:16.689319 21522 catalog_manager.cc:1347] Generated new cluster ID: 5696a8aed2044be98f5245bc2fcb6772
I20260925 15:24:16.689385 21522 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:24:16.690893 21508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:16.684810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57558 (local address 127.3.211.254:37305)
0925 15:24:16.685022 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:16.685032 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:16.685070 (+ 38us) server_negotiation.cc:407] Connection header received
0925 15:24:16.685170 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:16.685175 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:16.685232 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:16.685387 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:16.685616 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:16.687373 (+ 1757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:16.688312 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:16.689013 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:16.689164 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:16.690084 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:16.690088 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:16.690089 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:24:16.690318 (+ 229us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:16.690363 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:24:16.690380 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:16.690476 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:16.690549 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 15:24:16.690645 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":89,"threads_started":1}
I20260925 15:24:16.695820 21522 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:24:16.696645 21522 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:24:16.703886 21522 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2c3e28e73f84c9e9a440334270c2ef5: Generated new TSK 0
I20260925 15:24:16.704237 21522 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:24:16.708663 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.193:0
--local_ip_for_outbound_sockets=127.3.211.193
--webserver_interface=127.3.211.193
--webserver_port=0
--tserver_master_addrs=127.3.211.254:37305
--builtin_ntp_servers=127.3.211.212:39503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:24:16.796960 21526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:16.797235 21526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:16.797276 21526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:16.798648 21526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:16.798733 21526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:16.799034 21526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.193
I20260925 15:24:16.802167 21526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:39503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.211.193
--webserver_port=0
--tserver_master_addrs=127.3.211.254:37305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.193
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:24:16.802561 21526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:16.802986 21526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:16.808826 21531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:16.808858 21534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:16.808847 21532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:16.809573 21526 server_base.cc:1034] running on GCE node
I20260925 15:24:16.809852 21526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:16.810164 21526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:16.811345 21526 hybrid_clock.cc:648] HybridClock initialized: now 1790349856811321 us; error 40 us; skew 500 ppm
I20260925 15:24:16.813002 21526 webserver.cc:460] Webserver started at http://127.3.211.193:42797/ using document root <none> and password file <none>
I20260925 15:24:16.813300 21526 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:16.813359 21526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:16.813536 21526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:16.814728 21526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/ts-0/data/instance:
uuid: "09e9ccf001d64dada2c228f18d45bb88"
format_stamp: "Formatted at 2026-09-25 15:24:16 on dist-test-slave-4bk3"
server_key: "5831a86d845c209c8c618a86ab134202"
server_key_iv: "d50413eaf880cc6b29c623008f199855"
server_key_version: "encryptionkey@0"
I20260925 15:24:16.815341 21526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/ts-0/wal/instance:
uuid: "09e9ccf001d64dada2c228f18d45bb88"
format_stamp: "Formatted at 2026-09-25 15:24:16 on dist-test-slave-4bk3"
server_key: "5831a86d845c209c8c618a86ab134202"
server_key_iv: "d50413eaf880cc6b29c623008f199855"
server_key_version: "encryptionkey@0"
I20260925 15:24:16.816920 21526 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260925 15:24:16.817811 21540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:16.817963 21526 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:24:16.818078 21526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/ts-0/wal
uuid: "09e9ccf001d64dada2c228f18d45bb88"
format_stamp: "Formatted at 2026-09-25 15:24:16 on dist-test-slave-4bk3"
server_key: "5831a86d845c209c8c618a86ab134202"
server_key_iv: "d50413eaf880cc6b29c623008f199855"
server_key_version: "encryptionkey@0"
I20260925 15:24:16.818167 21526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:16.849617 21526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:16.850011 21526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:16.850296 21526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:16.850586 21526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:24:16.850955 21526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:24:16.850991 21526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:16.851015 21526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:24:16.851028 21526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:16.857553 21526 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.193:40063
I20260925 15:24:16.857592 21653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.193:40063 every 8 connection(s)
I20260925 15:24:16.858014 21526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:24:16.864032 21656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:16.858870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:37305 (local address 127.3.211.193:52257)
0925 15:24:16.859113 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:16.859149 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:16.859649 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:16.859963 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:16.859966 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:16.859975 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:24:16.860634 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:16.860640 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:16.861410 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:16.861411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:16.862230 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:16.862232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:16.862666 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:16.862667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:16.862792 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:16.863380 (+ 588us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:24:16.863380 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:24:16.863412 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:24:16.863437 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:24:16.863676 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:24:16.863680 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:24:16.863683 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:24:16.863745 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 15:24:16.863843 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":114,"threads_started":1}
I20260925 15:24:16.864013 21508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:16.858892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:52257 (local address 127.3.211.254:37305)
0925 15:24:16.858982 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:16.858984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:16.859241 (+ 257us) server_negotiation.cc:407] Connection header received
0925 15:24:16.859793 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:16.859795 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:16.859813 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:16.859882 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:16.860771 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:16.861271 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:16.862349 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:16.862546 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:16.862614 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:16.863583 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:16.863584 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:16.863585 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:24:16.863590 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:16.863605 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:24:16.863609 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:16.863664 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:16.863862 (+ 198us) server_negotiation.cc:299] Negotiation successful
0925 15:24:16.863921 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 15:24:16.865036 21654 heartbeater.cc:347] Connected to a master server at 127.3.211.254:37305
I20260925 15:24:16.865329 21654 heartbeater.cc:464] Registering TS with master...
I20260925 15:24:16.865361 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 21526
I20260925 15:24:16.865486 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/ts-0/wal/instance
I20260925 15:24:16.865628 21654 heartbeater.cc:511] Master 127.3.211.254:37305 requested a full tablet report, sending...
I20260925 15:24:16.865722 3919 external_mini_cluster.cc:1428] Setting key 721b8247ae760ab6a64ba0ac81396828
I20260925 15:24:16.866328 21466 ts_manager.cc:194] Registered new tserver with Master: 09e9ccf001d64dada2c228f18d45bb88 (127.3.211.193:40063)
I20260925 15:24:16.866978 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.194:0
--local_ip_for_outbound_sockets=127.3.211.194
--webserver_interface=127.3.211.194
--webserver_port=0
--tserver_master_addrs=127.3.211.254:37305
--builtin_ntp_servers=127.3.211.212:39503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 15:24:16.867340 21466 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.211.193:52257
W20260925 15:24:16.959638 21657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:16.959892 21657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:16.959918 21657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:16.961375 21657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:16.961454 21657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:16.961486 21657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.194
I20260925 15:24:16.963635 21657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:39503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.211.194
--webserver_port=0
--tserver_master_addrs=127.3.211.254:37305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.194
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:24:16.963894 21657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:16.964182 21657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:16.969702 21662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:16.969712 21665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:16.969961 21663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:16.970094 21657 server_base.cc:1034] running on GCE node
I20260925 15:24:16.970444 21657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:16.970752 21657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:16.972126 21657 hybrid_clock.cc:648] HybridClock initialized: now 1790349856972065 us; error 65 us; skew 500 ppm
I20260925 15:24:16.974152 21657 webserver.cc:460] Webserver started at http://127.3.211.194:43341/ using document root <none> and password file <none>
I20260925 15:24:16.974509 21657 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:16.974596 21657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:16.974762 21657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:16.975937 21657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/ts-1/data/instance:
uuid: "96433db4071e4010a39d7d91f61fb62c"
format_stamp: "Formatted at 2026-09-25 15:24:16 on dist-test-slave-4bk3"
server_key: "b1b8b225810d45e762bc5c68d7169a3c"
server_key_iv: "7e4704313fb1aa2c1d352e9e982b5acc"
server_key_version: "encryptionkey@0"
I20260925 15:24:16.976388 21657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/ts-1/wal/instance:
uuid: "96433db4071e4010a39d7d91f61fb62c"
format_stamp: "Formatted at 2026-09-25 15:24:16 on dist-test-slave-4bk3"
server_key: "b1b8b225810d45e762bc5c68d7169a3c"
server_key_iv: "7e4704313fb1aa2c1d352e9e982b5acc"
server_key_version: "encryptionkey@0"
I20260925 15:24:16.977931 21657 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:24:16.979018 21671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:16.979410 21657 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:24:16.979553 21657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/ts-1/wal
uuid: "96433db4071e4010a39d7d91f61fb62c"
format_stamp: "Formatted at 2026-09-25 15:24:16 on dist-test-slave-4bk3"
server_key: "b1b8b225810d45e762bc5c68d7169a3c"
server_key_iv: "7e4704313fb1aa2c1d352e9e982b5acc"
server_key_version: "encryptionkey@0"
I20260925 15:24:16.979648 21657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:17.002076 21657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:17.002488 21657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:17.002662 21657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:17.003050 21657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:24:17.003655 21657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:24:17.003876 21657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:17.003948 21657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:24:17.003978 21657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:17.010269 21657 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.194:40679
I20260925 15:24:17.010357 21784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.194:40679 every 8 connection(s)
I20260925 15:24:17.010715 21657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
I20260925 15:24:17.013744 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 21657
I20260925 15:24:17.013878 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/ts-1/wal/instance
I20260925 15:24:17.014109 3919 external_mini_cluster.cc:1428] Setting key 9b92980fab276fcd48967642fd3cb016
I20260925 15:24:17.015918 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.195:0
--local_ip_for_outbound_sockets=127.3.211.195
--webserver_interface=127.3.211.195
--webserver_port=0
--tserver_master_addrs=127.3.211.254:37305
--builtin_ntp_servers=127.3.211.212:39503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 15:24:17.024173 21508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:17.011592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:47787 (local address 127.3.211.254:37305)
0925 15:24:17.011688 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:17.011692 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:17.012147 (+ 455us) server_negotiation.cc:407] Connection header received
0925 15:24:17.012908 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:17.012910 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:17.012923 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:17.013011 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:17.014100 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:17.014750 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:17.016779 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:17.017291 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:17.017441 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:17.023825 (+ 6384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:17.023826 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:17.023827 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:24:17.023832 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:17.023859 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:24:17.023864 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:17.023925 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:17.024060 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 15:24:17.024100 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 15:24:17.024574 21787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:17.011582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:37305 (local address 127.3.211.194:47787)
0925 15:24:17.011928 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:17.011990 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:17.012724 (+ 734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:17.013153 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:17.013159 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:17.013170 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:24:17.013937 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:17.013945 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:17.014891 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:17.014893 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:17.016564 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:17.016570 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:17.017478 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:17.017480 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:17.017821 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:17.023539 (+ 5718us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:24:17.023541 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:24:17.023605 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:24:17.023652 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:24:17.023953 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:24:17.023959 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:24:17.023963 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:24:17.024045 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 15:24:17.024210 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":126,"threads_started":1}
I20260925 15:24:17.025377 21785 heartbeater.cc:347] Connected to a master server at 127.3.211.254:37305
I20260925 15:24:17.025478 21785 heartbeater.cc:464] Registering TS with master...
I20260925 15:24:17.025693 21785 heartbeater.cc:511] Master 127.3.211.254:37305 requested a full tablet report, sending...
I20260925 15:24:17.026129 21466 ts_manager.cc:194] Registered new tserver with Master: 96433db4071e4010a39d7d91f61fb62c (127.3.211.194:40679)
I20260925 15:24:17.026536 21466 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.211.194:47787
W20260925 15:24:17.106465 21788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:17.106750 21788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:17.106788 21788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:17.108045 21788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:17.108122 21788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:17.108166 21788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.195
I20260925 15:24:17.109773 21788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:39503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.211.195
--webserver_port=0
--tserver_master_addrs=127.3.211.254:37305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.195
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:24:17.110090 21788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:17.110443 21788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:17.115990 21796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:17.116011 21788 server_base.cc:1034] running on GCE node
W20260925 15:24:17.115991 21794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:17.115974 21793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:17.116443 21788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:17.116719 21788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:17.117919 21788 hybrid_clock.cc:648] HybridClock initialized: now 1790349857117893 us; error 37 us; skew 500 ppm
I20260925 15:24:17.119455 21788 webserver.cc:460] Webserver started at http://127.3.211.195:36429/ using document root <none> and password file <none>
I20260925 15:24:17.119686 21788 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:17.119740 21788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:17.119840 21788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:17.121155 21788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/ts-2/data/instance:
uuid: "0ac449ca6f224df286ce3385bf04f72c"
format_stamp: "Formatted at 2026-09-25 15:24:17 on dist-test-slave-4bk3"
server_key: "881c5c32c4e5cfcd288301eaf5c2d01c"
server_key_iv: "c4eabba7bec510ca62f7cf1ac9956bd3"
server_key_version: "encryptionkey@0"
I20260925 15:24:17.121523 21788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/ts-2/wal/instance:
uuid: "0ac449ca6f224df286ce3385bf04f72c"
format_stamp: "Formatted at 2026-09-25 15:24:17 on dist-test-slave-4bk3"
server_key: "881c5c32c4e5cfcd288301eaf5c2d01c"
server_key_iv: "c4eabba7bec510ca62f7cf1ac9956bd3"
server_key_version: "encryptionkey@0"
I20260925 15:24:17.123164 21788 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:24:17.124286 21802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:17.124534 21788 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:24:17.124639 21788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/ts-2/wal
uuid: "0ac449ca6f224df286ce3385bf04f72c"
format_stamp: "Formatted at 2026-09-25 15:24:17 on dist-test-slave-4bk3"
server_key: "881c5c32c4e5cfcd288301eaf5c2d01c"
server_key_iv: "c4eabba7bec510ca62f7cf1ac9956bd3"
server_key_version: "encryptionkey@0"
I20260925 15:24:17.124706 21788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:17.137818 21788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:17.138136 21788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:17.138252 21788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:17.138455 21788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:24:17.138827 21788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:24:17.138866 21788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:17.138895 21788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:24:17.138907 21788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:17.145869 21788 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.195:43331
I20260925 15:24:17.146971 21788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
I20260925 15:24:17.147122 21915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.195:43331 every 8 connection(s)
I20260925 15:24:17.152930 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 21788
I20260925 15:24:17.153081 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/ts-2/wal/instance
I20260925 15:24:17.153287 3919 external_mini_cluster.cc:1428] Setting key a2367618eecfe5e702a92bc0dfe8fa36
I20260925 15:24:17.153710 21508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:17.147998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:56727 (local address 127.3.211.254:37305)
0925 15:24:17.148057 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:17.148059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:17.148290 (+ 231us) server_negotiation.cc:407] Connection header received
0925 15:24:17.148892 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:17.148894 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:17.148906 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:17.148975 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:17.149749 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:17.150269 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:17.151442 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:17.151697 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:17.151812 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:17.153268 (+ 1456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:17.153270 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:17.153271 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:24:17.153278 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:17.153299 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:24:17.153304 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:17.153379 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:17.153563 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 15:24:17.153615 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 15:24:17.153936 21918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:17.147864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:37305 (local address 127.3.211.195:56727)
0925 15:24:17.148138 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:17.148174 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:17.148751 (+ 577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:17.149081 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:17.149085 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:17.149094 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:24:17.149634 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:17.149638 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:17.150414 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:17.150415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:17.151258 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:17.151261 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:17.151886 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:17.151888 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:17.152211 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:17.153065 (+ 854us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:24:17.153065 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:24:17.153130 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:24:17.153167 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:24:17.153390 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:24:17.153396 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:24:17.153399 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:24:17.153529 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 15:24:17.153669 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":134,"threads_started":1}
I20260925 15:24:17.154500 21916 heartbeater.cc:347] Connected to a master server at 127.3.211.254:37305
I20260925 15:24:17.154603 21916 heartbeater.cc:464] Registering TS with master...
I20260925 15:24:17.154770 21916 heartbeater.cc:511] Master 127.3.211.254:37305 requested a full tablet report, sending...
I20260925 15:24:17.155220 21466 ts_manager.cc:194] Registered new tserver with Master: 0ac449ca6f224df286ce3385bf04f72c (127.3.211.195:43331)
I20260925 15:24:17.155616 21466 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.211.195:56727
I20260925 15:24:17.165778 3919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:24:17.171703 21508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:17.168082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57570 (local address 127.3.211.254:37305)
0925 15:24:17.168185 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:17.168189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:17.168302 (+ 113us) server_negotiation.cc:407] Connection header received
0925 15:24:17.168445 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:17.168446 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:17.168463 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:17.168548 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:17.168745 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:17.169259 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:17.170110 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:17.170311 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:17.170400 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:17.171395 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:17.171397 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:17.171397 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:24:17.171405 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:17.171426 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:24:17.171437 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:17.171501 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:17.171565 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 15:24:17.171596 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 15:24:17.173854 21466 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57570:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:24:17.175424 21466 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:24:17.185464 21924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:17.179138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:40063 (local address 127.0.0.1:50568)
0925 15:24:17.179579 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:17.179587 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:17.179625 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:17.180003 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:17.180005 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:17.180008 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:24:17.180085 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:17.180088 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:17.181198 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:17.181200 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:17.182602 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:17.182606 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:17.184683 (+ 2077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:17.184684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:17.185023 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:17.185050 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 15:24:17.185131 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 15:24:17.185230 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"mutex_wait_us":1,"thread_start_us":73,"threads_started":1}
I20260925 15:24:17.185736 21925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:17.178907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.194:40679 (local address 127.0.0.1:60344)
0925 15:24:17.179438 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:17.179448 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:17.179491 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:17.180080 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:17.180082 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:17.180085 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:24:17.180135 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:17.180137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:17.181798 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:17.181799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:17.182908 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:17.182911 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:17.184684 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:17.184685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:17.185140 (+ 455us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:17.185143 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:24:17.185190 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 15:24:17.185454 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"mutex_wait_us":6,"thread_start_us":86,"threads_started":1}
I20260925 15:24:17.185866 21926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:17.179113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60344 (local address 127.3.211.194:40679)
0925 15:24:17.179624 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:17.179629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:17.179640 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:24:17.179680 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:17.179682 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:17.179698 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:17.179958 (+ 260us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:24:17.180226 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:17.181689 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:17.183004 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:17.184476 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:17.184552 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:17.184642 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:17.185568 (+ 926us) server_negotiation.cc:299] Negotiation successful
0925 15:24:17.185652 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":91,"threads_started":1}
I20260925 15:24:17.186100 21929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:17.179286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50568 (local address 127.3.211.193:40063)
0925 15:24:17.179847 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:17.179850 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:17.179858 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:24:17.179890 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:17.179891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:17.179902 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:17.179994 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:24:17.180173 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:17.181105 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:17.182718 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:17.183678 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:17.183753 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:17.183864 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:17.185930 (+ 2066us) server_negotiation.cc:299] Negotiation successful
0925 15:24:17.185990 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":115,"threads_started":1}
I20260925 15:24:17.186825 21719 tablet_service.cc:1467] Processing CreateTablet for tablet a2148f4eb9ba479e8ff9e4a18625df03 (DEFAULT_TABLE table=test-table [id=24fa831db78c4508a49eee3c06802ccf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:24:17.187434 21719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2148f4eb9ba479e8ff9e4a18625df03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:17.187506 21588 tablet_service.cc:1467] Processing CreateTablet for tablet a2148f4eb9ba479e8ff9e4a18625df03 (DEFAULT_TABLE table=test-table [id=24fa831db78c4508a49eee3c06802ccf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:24:17.187997 21588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2148f4eb9ba479e8ff9e4a18625df03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:17.188454 21927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:17.178907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.195:43331 (local address 127.0.0.1:50060)
0925 15:24:17.179341 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:17.179390 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:17.179470 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:17.179737 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:17.179740 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:17.179829 (+ 89us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:24:17.179996 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:17.180009 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:17.181819 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:17.181821 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:17.183971 (+ 2150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:17.183976 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:17.188123 (+ 4147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:17.188125 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:17.188245 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:17.188251 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 15:24:17.188287 (+ 36us) client_negotiation.cc:240] Negotiation successful
0925 15:24:17.188366 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":2,"thread_start_us":87,"threads_started":1}
I20260925 15:24:17.188948 21928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:17.179278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50060 (local address 127.3.211.195:43331)
0925 15:24:17.179459 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:17.179465 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:17.179478 (+ 13us) server_negotiation.cc:407] Connection header received
0925 15:24:17.179567 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:17.179570 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:17.179587 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:17.179704 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:24:17.180223 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:17.181689 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:17.184132 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:17.187990 (+ 3858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:17.188647 (+ 657us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:17.188764 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:17.188795 (+ 31us) server_negotiation.cc:299] Negotiation successful
0925 15:24:17.188864 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":84,"threads_started":1}
I20260925 15:24:17.192889 21930 tablet_bootstrap.cc:492] T a2148f4eb9ba479e8ff9e4a18625df03 P 96433db4071e4010a39d7d91f61fb62c: Bootstrap starting.
I20260925 15:24:17.193248 21931 tablet_bootstrap.cc:492] T a2148f4eb9ba479e8ff9e4a18625df03 P 09e9ccf001d64dada2c228f18d45bb88: Bootstrap starting.
I20260925 15:24:17.193998 21931 tablet_bootstrap.cc:654] T a2148f4eb9ba479e8ff9e4a18625df03 P 09e9ccf001d64dada2c228f18d45bb88: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:17.194161 21930 tablet_bootstrap.cc:654] T a2148f4eb9ba479e8ff9e4a18625df03 P 96433db4071e4010a39d7d91f61fb62c: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:17.194382 21931 log.cc:824] T a2148f4eb9ba479e8ff9e4a18625df03 P 09e9ccf001d64dada2c228f18d45bb88: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:17.194545 21930 log.cc:824] T a2148f4eb9ba479e8ff9e4a18625df03 P 96433db4071e4010a39d7d91f61fb62c: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:17.195279 21931 tablet_bootstrap.cc:492] T a2148f4eb9ba479e8ff9e4a18625df03 P 09e9ccf001d64dada2c228f18d45bb88: No bootstrap required, opened a new log
I20260925 15:24:17.195356 21931 ts_tablet_manager.cc:1397] T a2148f4eb9ba479e8ff9e4a18625df03 P 09e9ccf001d64dada2c228f18d45bb88: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:24:17.195597 21930 tablet_bootstrap.cc:492] T a2148f4eb9ba479e8ff9e4a18625df03 P 96433db4071e4010a39d7d91f61fb62c: No bootstrap required, opened a new log
I20260925 15:24:17.195686 21930 ts_tablet_manager.cc:1397] T a2148f4eb9ba479e8ff9e4a18625df03 P 96433db4071e4010a39d7d91f61fb62c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 15:24:17.195622 21850 tablet_service.cc:1467] Processing CreateTablet for tablet a2148f4eb9ba479e8ff9e4a18625df03 (DEFAULT_TABLE table=test-table [id=24fa831db78c4508a49eee3c06802ccf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:24:17.196051 21850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2148f4eb9ba479e8ff9e4a18625df03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:17.197916 21930 raft_consensus.cc:348] T a2148f4eb9ba479e8ff9e4a18625df03 P 96433db4071e4010a39d7d91f61fb62c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ac449ca6f224df286ce3385bf04f72c" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 43331 } } peers { permanent_uuid: "09e9ccf001d64dada2c228f18d45bb88" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 40063 } } peers { permanent_uuid: "96433db4071e4010a39d7d91f61fb62c" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40679 } }
I20260925 15:24:17.198184 21931 raft_consensus.cc:348] T a2148f4eb9ba479e8ff9e4a18625df03 P 09e9ccf001d64dada2c228f18d45bb88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ac449ca6f224df286ce3385bf04f72c" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 43331 } } peers { permanent_uuid: "09e9ccf001d64dada2c228f18d45bb88" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 40063 } } peers { permanent_uuid: "96433db4071e4010a39d7d91f61fb62c" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40679 } }
I20260925 15:24:17.198321 21930 raft_consensus.cc:374] T a2148f4eb9ba479e8ff9e4a18625df03 P 96433db4071e4010a39d7d91f61fb62c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:24:17.198372 21930 raft_consensus.cc:729] T a2148f4eb9ba479e8ff9e4a18625df03 P 96433db4071e4010a39d7d91f61fb62c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96433db4071e4010a39d7d91f61fb62c, State: Initialized, Role: FOLLOWER
I20260925 15:24:17.198376 21931 raft_consensus.cc:374] T a2148f4eb9ba479e8ff9e4a18625df03 P 09e9ccf001d64dada2c228f18d45bb88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:24:17.198413 21931 raft_consensus.cc:729] T a2148f4eb9ba479e8ff9e4a18625df03 P 09e9ccf001d64dada2c228f18d45bb88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09e9ccf001d64dada2c228f18d45bb88, State: Initialized, Role: FOLLOWER
I20260925 15:24:17.198506 21930 consensus_queue.cc:260] T a2148f4eb9ba479e8ff9e4a18625df03 P 96433db4071e4010a39d7d91f61fb62c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ac449ca6f224df286ce3385bf04f72c" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 43331 } } peers { permanent_uuid: "09e9ccf001d64dada2c228f18d45bb88" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 40063 } } peers { permanent_uuid: "96433db4071e4010a39d7d91f61fb62c" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40679 } }
I20260925 15:24:17.198532 21931 consensus_queue.cc:260] T a2148f4eb9ba479e8ff9e4a18625df03 P 09e9ccf001d64dada2c228f18d45bb88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ac449ca6f224df286ce3385bf04f72c" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 43331 } } peers { permanent_uuid: "09e9ccf001d64dada2c228f18d45bb88" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 40063 } } peers { permanent_uuid: "96433db4071e4010a39d7d91f61fb62c" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40679 } }
I20260925 15:24:17.198827 21785 heartbeater.cc:503] Master 127.3.211.254:37305 was elected leader, sending a full tablet report...
I20260925 15:24:17.199038 21931 ts_tablet_manager.cc:1428] T a2148f4eb9ba479e8ff9e4a18625df03 P 09e9ccf001d64dada2c228f18d45bb88: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260925 15:24:17.199222 21930 ts_tablet_manager.cc:1428] T a2148f4eb9ba479e8ff9e4a18625df03 P 96433db4071e4010a39d7d91f61fb62c: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260925 15:24:17.199342 21654 heartbeater.cc:503] Master 127.3.211.254:37305 was elected leader, sending a full tablet report...
I20260925 15:24:17.202203 21936 tablet_bootstrap.cc:492] T a2148f4eb9ba479e8ff9e4a18625df03 P 0ac449ca6f224df286ce3385bf04f72c: Bootstrap starting.
I20260925 15:24:17.203020 21936 tablet_bootstrap.cc:654] T a2148f4eb9ba479e8ff9e4a18625df03 P 0ac449ca6f224df286ce3385bf04f72c: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:17.203383 21936 log.cc:824] T a2148f4eb9ba479e8ff9e4a18625df03 P 0ac449ca6f224df286ce3385bf04f72c: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:17.205042 21936 tablet_bootstrap.cc:492] T a2148f4eb9ba479e8ff9e4a18625df03 P 0ac449ca6f224df286ce3385bf04f72c: No bootstrap required, opened a new log
I20260925 15:24:17.205142 21936 ts_tablet_manager.cc:1397] T a2148f4eb9ba479e8ff9e4a18625df03 P 0ac449ca6f224df286ce3385bf04f72c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 15:24:17.207826 21936 raft_consensus.cc:348] T a2148f4eb9ba479e8ff9e4a18625df03 P 0ac449ca6f224df286ce3385bf04f72c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ac449ca6f224df286ce3385bf04f72c" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 43331 } } peers { permanent_uuid: "09e9ccf001d64dada2c228f18d45bb88" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 40063 } } peers { permanent_uuid: "96433db4071e4010a39d7d91f61fb62c" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40679 } }
I20260925 15:24:17.208066 21936 raft_consensus.cc:374] T a2148f4eb9ba479e8ff9e4a18625df03 P 0ac449ca6f224df286ce3385bf04f72c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:24:17.208108 21936 raft_consensus.cc:729] T a2148f4eb9ba479e8ff9e4a18625df03 P 0ac449ca6f224df286ce3385bf04f72c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ac449ca6f224df286ce3385bf04f72c, State: Initialized, Role: FOLLOWER
I20260925 15:24:17.208248 21936 consensus_queue.cc:260] T a2148f4eb9ba479e8ff9e4a18625df03 P 0ac449ca6f224df286ce3385bf04f72c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ac449ca6f224df286ce3385bf04f72c" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 43331 } } peers { permanent_uuid: "09e9ccf001d64dada2c228f18d45bb88" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 40063 } } peers { permanent_uuid: "96433db4071e4010a39d7d91f61fb62c" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40679 } }
I20260925 15:24:17.208590 21936 ts_tablet_manager.cc:1428] T a2148f4eb9ba479e8ff9e4a18625df03 P 0ac449ca6f224df286ce3385bf04f72c: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260925 15:24:17.208643 21916 heartbeater.cc:503] Master 127.3.211.254:37305 was elected leader, sending a full tablet report...
W20260925 15:24:17.261613 21786 tablet.cc:2369] T a2148f4eb9ba479e8ff9e4a18625df03 P 96433db4071e4010a39d7d91f61fb62c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:24:17.358795 21655 tablet.cc:2369] T a2148f4eb9ba479e8ff9e4a18625df03 P 09e9ccf001d64dada2c228f18d45bb88: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:24:17.397771 21917 tablet.cc:2369] T a2148f4eb9ba479e8ff9e4a18625df03 P 0ac449ca6f224df286ce3385bf04f72c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:24:17.603794 21934 raft_consensus.cc:482] T a2148f4eb9ba479e8ff9e4a18625df03 P 96433db4071e4010a39d7d91f61fb62c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:24:17.604009 21934 raft_consensus.cc:504] T a2148f4eb9ba479e8ff9e4a18625df03 P 96433db4071e4010a39d7d91f61fb62c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ac449ca6f224df286ce3385bf04f72c" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 43331 } } peers { permanent_uuid: "09e9ccf001d64dada2c228f18d45bb88" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 40063 } } peers { permanent_uuid: "96433db4071e4010a39d7d91f61fb62c" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40679 } }
I20260925 15:24:17.604835 21934 leader_election.cc:288] T a2148f4eb9ba479e8ff9e4a18625df03 P 96433db4071e4010a39d7d91f61fb62c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ac449ca6f224df286ce3385bf04f72c (127.3.211.195:43331), 09e9ccf001d64dada2c228f18d45bb88 (127.3.211.193:40063)
I20260925 15:24:17.605888 21935 raft_consensus.cc:482] T a2148f4eb9ba479e8ff9e4a18625df03 P 09e9ccf001d64dada2c228f18d45bb88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:24:17.606021 21935 raft_consensus.cc:504] T a2148f4eb9ba479e8ff9e4a18625df03 P 09e9ccf001d64dada2c228f18d45bb88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ac449ca6f224df286ce3385bf04f72c" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 43331 } } peers { permanent_uuid: "09e9ccf001d64dada2c228f18d45bb88" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 40063 } } peers { permanent_uuid: "96433db4071e4010a39d7d91f61fb62c" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40679 } }
I20260925 15:24:17.606468 21935 leader_election.cc:288] T a2148f4eb9ba479e8ff9e4a18625df03 P 09e9ccf001d64dada2c228f18d45bb88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ac449ca6f224df286ce3385bf04f72c (127.3.211.195:43331), 96433db4071e4010a39d7d91f61fb62c (127.3.211.194:40679)
I20260925 15:24:17.609540 21928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:17.605385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:47325 (local address 127.3.211.195:43331)
0925 15:24:17.605508 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:17.605511 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:17.605631 (+ 120us) server_negotiation.cc:407] Connection header received
0925 15:24:17.605754 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:17.605755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:17.605772 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:17.605805 (+ 33us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:24:17.606170 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:17.606915 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:17.608587 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:17.609144 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:17.609193 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:17.609202 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:17.609416 (+ 214us) server_negotiation.cc:299] Negotiation successful
0925 15:24:17.609465 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 15:24:17.609980 21939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:17.605292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.195:43331 (local address 127.3.211.194:47325)
0925 15:24:17.605497 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:17.605516 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:17.605657 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:17.605898 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:17.605899 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:17.605902 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:24:17.606046 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:17.606051 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:17.607116 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:17.607117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:17.608480 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:17.608482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:17.609273 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:17.609273 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:17.609335 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:17.609339 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:24:17.609844 (+ 505us) client_negotiation.cc:240] Negotiation successful
0925 15:24:17.609892 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":1,"thread_start_us":99,"threads_started":1}
I20260925 15:24:17.610013 21929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:17.605705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:39683 (local address 127.3.211.193:40063)
0925 15:24:17.605787 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:17.605789 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:17.605801 (+ 12us) server_negotiation.cc:407] Connection header received
0925 15:24:17.605837 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:17.605838 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:17.605852 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:17.605932 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:24:17.606193 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:17.607017 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:17.608897 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:17.609507 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:17.609575 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:17.609583 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:17.609890 (+ 307us) server_negotiation.cc:299] Negotiation successful
0925 15:24:17.609930 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":58}
I20260925 15:24:17.610280 21940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:17.605539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:40063 (local address 127.3.211.194:39683)
0925 15:24:17.605672 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:17.605686 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:17.605757 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:17.605971 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:17.605972 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:17.605974 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:24:17.606094 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:17.606096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:17.607162 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:17.607163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:17.608799 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:17.608803 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:17.609730 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:17.609731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:17.609808 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:17.609813 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:24:17.610131 (+ 318us) client_negotiation.cc:240] Negotiation successful
0925 15:24:17.610189 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":47,"threads_started":1}
I20260925 15:24:17.610390 21926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:17.606850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:35121 (local address 127.3.211.194:40679)
0925 15:24:17.606889 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:17.606891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:17.606990 (+ 99us) server_negotiation.cc:407] Connection header received
0925 15:24:17.607091 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:17.607092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:17.607104 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:17.607134 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:24:17.607333 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:17.608043 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:17.609508 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:17.610017 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:17.610068 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:17.610076 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:17.610294 (+ 218us) server_negotiation.cc:299] Negotiation successful
0925 15:24:17.610334 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 15:24:17.610575 21941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:17.606700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.194:40679 (local address 127.3.211.193:35121)
0925 15:24:17.606933 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:17.606944 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:17.607019 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:17.607192 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:17.607193 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:17.607195 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:24:17.607262 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:17.607264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:17.608159 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:17.608160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:17.609410 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:17.609412 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:17.610159 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:17.610160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:17.610222 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:17.610225 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:24:17.610432 (+ 207us) client_negotiation.cc:240] Negotiation successful
0925 15:24:17.610479 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":55,"threads_started":1}
I20260925 15:24:17.610630 21870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2148f4eb9ba479e8ff9e4a18625df03" candidate_uuid: "96433db4071e4010a39d7d91f61fb62c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ac449ca6f224df286ce3385bf04f72c" is_pre_election: true
I20260925 15:24:17.610728 21608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2148f4eb9ba479e8ff9e4a18625df03" candidate_uuid: "96433db4071e4010a39d7d91f61fb62c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09e9ccf001d64dada2c228f18d45bb88" is_pre_election: true
I20260925 15:24:17.610787 21870 raft_consensus.cc:2454] T a2148f4eb9ba479e8ff9e4a18625df03 P 0ac449ca6f224df286ce3385bf04f72c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96433db4071e4010a39d7d91f61fb62c in term 0.
I20260925 15:24:17.610802 21608 raft_consensus.cc:2454] T a2148f4eb9ba479e8ff9e4a18625df03 P 09e9ccf001d64dada2c228f18d45bb88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96433db4071e4010a39d7d91f61fb62c in term 0.
I20260925 15:24:17.610858 21739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2148f4eb9ba479e8ff9e4a18625df03" candidate_uuid: "09e9ccf001d64dada2c228f18d45bb88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96433db4071e4010a39d7d91f61fb62c" is_pre_election: true
I20260925 15:24:17.610993 21674 leader_election.cc:302] T a2148f4eb9ba479e8ff9e4a18625df03 P 96433db4071e4010a39d7d91f61fb62c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ac449ca6f224df286ce3385bf04f72c, 96433db4071e4010a39d7d91f61fb62c; no voters:
I20260925 15:24:17.610998 21739 raft_consensus.cc:2454] T a2148f4eb9ba479e8ff9e4a18625df03 P 96433db4071e4010a39d7d91f61fb62c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09e9ccf001d64dada2c228f18d45bb88 in term 0.
I20260925 15:24:17.611413 21934 raft_consensus.cc:2781] T a2148f4eb9ba479e8ff9e4a18625df03 P 96433db4071e4010a39d7d91f61fb62c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:24:17.611526 21934 raft_consensus.cc:482] T a2148f4eb9ba479e8ff9e4a18625df03 P 96433db4071e4010a39d7d91f61fb62c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:24:17.611600 21934 raft_consensus.cc:3037] T a2148f4eb9ba479e8ff9e4a18625df03 P 96433db4071e4010a39d7d91f61fb62c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:17.611720 21942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:17.606744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.195:43331 (local address 127.3.211.193:35691)
0925 15:24:17.607078 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:17.607089 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:17.607135 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:17.607373 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:17.607374 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:17.607376 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:24:17.607563 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:17.607569 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:17.608851 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:17.608852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:17.610088 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:17.610091 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:17.610960 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:17.610961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:17.611029 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:17.611279 (+ 250us) client_negotiation.cc:769] Sending connection context
0925 15:24:17.611390 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 15:24:17.611478 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":57,"threads_started":1}
I20260925 15:24:17.611720 21544 leader_election.cc:302] T a2148f4eb9ba479e8ff9e4a18625df03 P 09e9ccf001d64dada2c228f18d45bb88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09e9ccf001d64dada2c228f18d45bb88, 96433db4071e4010a39d7d91f61fb62c; no voters:
I20260925 15:24:17.611970 21935 raft_consensus.cc:2781] T a2148f4eb9ba479e8ff9e4a18625df03 P 09e9ccf001d64dada2c228f18d45bb88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:24:17.612030 21935 raft_consensus.cc:482] T a2148f4eb9ba479e8ff9e4a18625df03 P 09e9ccf001d64dada2c228f18d45bb88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:24:17.612068 21935 raft_consensus.cc:3037] T a2148f4eb9ba479e8ff9e4a18625df03 P 09e9ccf001d64dada2c228f18d45bb88 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:17.612123 21943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:17.606931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:35691 (local address 127.3.211.195:43331)
0925 15:24:17.607250 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:17.607252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:17.607258 (+ 6us) server_negotiation.cc:407] Connection header received
0925 15:24:17.607278 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:17.607279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:17.607287 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:17.607699 (+ 412us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:24:17.607749 (+ 50us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:17.608726 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:17.610193 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:17.610861 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:17.611673 (+ 812us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:17.611685 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:17.611706 (+ 21us) server_negotiation.cc:299] Negotiation successful
0925 15:24:17.611751 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":94,"threads_started":1}
I20260925 15:24:17.612296 21870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2148f4eb9ba479e8ff9e4a18625df03" candidate_uuid: "09e9ccf001d64dada2c228f18d45bb88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ac449ca6f224df286ce3385bf04f72c" is_pre_election: true
I20260925 15:24:17.612397 21870 raft_consensus.cc:2454] T a2148f4eb9ba479e8ff9e4a18625df03 P 0ac449ca6f224df286ce3385bf04f72c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09e9ccf001d64dada2c228f18d45bb88 in term 0.
I20260925 15:24:17.612828 21934 raft_consensus.cc:504] T a2148f4eb9ba479e8ff9e4a18625df03 P 96433db4071e4010a39d7d91f61fb62c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ac449ca6f224df286ce3385bf04f72c" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 43331 } } peers { permanent_uuid: "09e9ccf001d64dada2c228f18d45bb88" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 40063 } } peers { permanent_uuid: "96433db4071e4010a39d7d91f61fb62c" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40679 } }
I20260925 15:24:17.612974 21934 leader_election.cc:288] T a2148f4eb9ba479e8ff9e4a18625df03 P 96433db4071e4010a39d7d91f61fb62c [CANDIDATE]: Term 1 election: Requested vote from peers 0ac449ca6f224df286ce3385bf04f72c (127.3.211.195:43331), 09e9ccf001d64dada2c228f18d45bb88 (127.3.211.193:40063)
I20260925 15:24:17.613126 21870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2148f4eb9ba479e8ff9e4a18625df03" candidate_uuid: "96433db4071e4010a39d7d91f61fb62c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ac449ca6f224df286ce3385bf04f72c"
I20260925 15:24:17.613205 21870 raft_consensus.cc:3037] T a2148f4eb9ba479e8ff9e4a18625df03 P 0ac449ca6f224df286ce3385bf04f72c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:17.613250 21608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2148f4eb9ba479e8ff9e4a18625df03" candidate_uuid: "96433db4071e4010a39d7d91f61fb62c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09e9ccf001d64dada2c228f18d45bb88"
I20260925 15:24:17.613375 21935 raft_consensus.cc:504] T a2148f4eb9ba479e8ff9e4a18625df03 P 09e9ccf001d64dada2c228f18d45bb88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ac449ca6f224df286ce3385bf04f72c" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 43331 } } peers { permanent_uuid: "09e9ccf001d64dada2c228f18d45bb88" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 40063 } } peers { permanent_uuid: "96433db4071e4010a39d7d91f61fb62c" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40679 } }
I20260925 15:24:17.613516 21608 raft_consensus.cc:2379] T a2148f4eb9ba479e8ff9e4a18625df03 P 09e9ccf001d64dada2c228f18d45bb88 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 96433db4071e4010a39d7d91f61fb62c in current term 1: Already voted for candidate 09e9ccf001d64dada2c228f18d45bb88 in this term.
I20260925 15:24:17.613538 21935 leader_election.cc:288] T a2148f4eb9ba479e8ff9e4a18625df03 P 09e9ccf001d64dada2c228f18d45bb88 [CANDIDATE]: Term 1 election: Requested vote from peers 0ac449ca6f224df286ce3385bf04f72c (127.3.211.195:43331), 96433db4071e4010a39d7d91f61fb62c (127.3.211.194:40679)
I20260925 15:24:17.613869 21869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2148f4eb9ba479e8ff9e4a18625df03" candidate_uuid: "09e9ccf001d64dada2c228f18d45bb88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ac449ca6f224df286ce3385bf04f72c"
I20260925 15:24:17.613988 21739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2148f4eb9ba479e8ff9e4a18625df03" candidate_uuid: "09e9ccf001d64dada2c228f18d45bb88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96433db4071e4010a39d7d91f61fb62c"
I20260925 15:24:17.614074 21739 raft_consensus.cc:2379] T a2148f4eb9ba479e8ff9e4a18625df03 P 96433db4071e4010a39d7d91f61fb62c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 09e9ccf001d64dada2c228f18d45bb88 in current term 1: Already voted for candidate 96433db4071e4010a39d7d91f61fb62c in this term.
I20260925 15:24:17.614296 21870 raft_consensus.cc:2454] T a2148f4eb9ba479e8ff9e4a18625df03 P 0ac449ca6f224df286ce3385bf04f72c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96433db4071e4010a39d7d91f61fb62c in term 1.
I20260925 15:24:17.614282 21544 leader_election.cc:302] T a2148f4eb9ba479e8ff9e4a18625df03 P 09e9ccf001d64dada2c228f18d45bb88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09e9ccf001d64dada2c228f18d45bb88; no voters: 0ac449ca6f224df286ce3385bf04f72c, 96433db4071e4010a39d7d91f61fb62c
I20260925 15:24:17.614500 21674 leader_election.cc:302] T a2148f4eb9ba479e8ff9e4a18625df03 P 96433db4071e4010a39d7d91f61fb62c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ac449ca6f224df286ce3385bf04f72c, 96433db4071e4010a39d7d91f61fb62c; no voters: 09e9ccf001d64dada2c228f18d45bb88
I20260925 15:24:17.614521 21935 raft_consensus.cc:2726] T a2148f4eb9ba479e8ff9e4a18625df03 P 09e9ccf001d64dada2c228f18d45bb88 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260925 15:24:17.614636 21934 raft_consensus.cc:2781] T a2148f4eb9ba479e8ff9e4a18625df03 P 96433db4071e4010a39d7d91f61fb62c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:24:17.614831 21934 raft_consensus.cc:686] T a2148f4eb9ba479e8ff9e4a18625df03 P 96433db4071e4010a39d7d91f61fb62c [term 1 LEADER]: Becoming Leader. State: Replica: 96433db4071e4010a39d7d91f61fb62c, State: Running, Role: LEADER
I20260925 15:24:17.614948 21934 consensus_queue.cc:237] T a2148f4eb9ba479e8ff9e4a18625df03 P 96433db4071e4010a39d7d91f61fb62c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ac449ca6f224df286ce3385bf04f72c" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 43331 } } peers { permanent_uuid: "09e9ccf001d64dada2c228f18d45bb88" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 40063 } } peers { permanent_uuid: "96433db4071e4010a39d7d91f61fb62c" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40679 } }
I20260925 15:24:17.615888 21466 catalog_manager.cc:5495] T a2148f4eb9ba479e8ff9e4a18625df03 P 96433db4071e4010a39d7d91f61fb62c reported cstate change: term changed from 0 to 1, leader changed from <none> to 96433db4071e4010a39d7d91f61fb62c (127.3.211.194). New cstate: current_term: 1 leader_uuid: "96433db4071e4010a39d7d91f61fb62c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ac449ca6f224df286ce3385bf04f72c" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 43331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09e9ccf001d64dada2c228f18d45bb88" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 40063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96433db4071e4010a39d7d91f61fb62c" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40679 } health_report { overall_health: HEALTHY } } }
I20260925 15:24:17.636902 21926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:17.634189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60360 (local address 127.3.211.194:40679)
0925 15:24:17.634258 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:17.634260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:17.634277 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:24:17.634324 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:17.634325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:17.634350 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:17.634423 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:24:17.634587 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:17.635067 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:17.635982 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:17.636184 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:17.636252 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:17.636510 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:24:17.636625 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:24:17.636696 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:17.636815 (+ 119us) server_negotiation.cc:299] Negotiation successful
0925 15:24:17.636837 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 15:24:17.638881 21870 raft_consensus.cc:1261] T a2148f4eb9ba479e8ff9e4a18625df03 P 0ac449ca6f224df286ce3385bf04f72c [term 1 FOLLOWER]: Refusing update from remote peer 96433db4071e4010a39d7d91f61fb62c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:24:17.638897 21608 raft_consensus.cc:1261] T a2148f4eb9ba479e8ff9e4a18625df03 P 09e9ccf001d64dada2c228f18d45bb88 [term 1 FOLLOWER]: Refusing update from remote peer 96433db4071e4010a39d7d91f61fb62c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:24:17.639179 21934 consensus_queue.cc:1035] T a2148f4eb9ba479e8ff9e4a18625df03 P 96433db4071e4010a39d7d91f61fb62c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ac449ca6f224df286ce3385bf04f72c" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 43331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:24:17.639370 21944 consensus_queue.cc:1035] T a2148f4eb9ba479e8ff9e4a18625df03 P 96433db4071e4010a39d7d91f61fb62c [LEADER]: Connected to new peer: Peer: permanent_uuid: "09e9ccf001d64dada2c228f18d45bb88" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 40063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:24:17.642017 21947 mvcc.cc:205] Tried to move back new op lower bound from 7333273016885776384 to 7333273016791552000. Current Snapshot: MvccSnapshot[applied={T|T < 7333273016885776384}]
I20260925 15:24:17.642217 21949 mvcc.cc:205] Tried to move back new op lower bound from 7333273016885776384 to 7333273016791552000. Current Snapshot: MvccSnapshot[applied={T|T < 7333273016885776384}]
I20260925 15:24:17.643069 21948 mvcc.cc:205] Tried to move back new op lower bound from 7333273016885776384 to 7333273016791552000. Current Snapshot: MvccSnapshot[applied={T|T < 7333273016885776384 or (T in {7333273016885776384})}]
I20260925 15:24:17.645730 21465 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:24:17.645854 21465 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:24:17.646674 21465 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e2c3e28e73f84c9e9a440334270c2ef5: Sending DeleteTablet for 3 replicas of tablet a2148f4eb9ba479e8ff9e4a18625df03
I20260925 15:24:17.647073 21588 tablet_service.cc:1514] Processing DeleteTablet for tablet a2148f4eb9ba479e8ff9e4a18625df03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:24:17 UTC) from {username='slave'} at 127.0.0.1:50568
I20260925 15:24:17.647066 21850 tablet_service.cc:1514] Processing DeleteTablet for tablet a2148f4eb9ba479e8ff9e4a18625df03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:24:17 UTC) from {username='slave'} at 127.0.0.1:50060
I20260925 15:24:17.647485 21958 tablet_replica.cc:330] T a2148f4eb9ba479e8ff9e4a18625df03 P 09e9ccf001d64dada2c228f18d45bb88: stopping tablet replica
I20260925 15:24:17.647502 21719 tablet_service.cc:1514] Processing DeleteTablet for tablet a2148f4eb9ba479e8ff9e4a18625df03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:24:17 UTC) from {username='slave'} at 127.0.0.1:60344
I20260925 15:24:17.647610 21958 raft_consensus.cc:2229] T a2148f4eb9ba479e8ff9e4a18625df03 P 09e9ccf001d64dada2c228f18d45bb88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:24:17.647743 21958 raft_consensus.cc:2258] T a2148f4eb9ba479e8ff9e4a18625df03 P 09e9ccf001d64dada2c228f18d45bb88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:24:17.648020 21959 tablet_replica.cc:330] T a2148f4eb9ba479e8ff9e4a18625df03 P 96433db4071e4010a39d7d91f61fb62c: stopping tablet replica
I20260925 15:24:17.648142 21959 raft_consensus.cc:2229] T a2148f4eb9ba479e8ff9e4a18625df03 P 96433db4071e4010a39d7d91f61fb62c [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:24:17.648180 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 21526
I20260925 15:24:17.648279 21958 ts_tablet_manager.cc:1905] T a2148f4eb9ba479e8ff9e4a18625df03 P 09e9ccf001d64dada2c228f18d45bb88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:24:17.648311 21959 raft_consensus.cc:2258] T a2148f4eb9ba479e8ff9e4a18625df03 P 96433db4071e4010a39d7d91f61fb62c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:24:17.649055 21957 tablet_replica.cc:330] T a2148f4eb9ba479e8ff9e4a18625df03 P 0ac449ca6f224df286ce3385bf04f72c: stopping tablet replica
I20260925 15:24:17.649185 21957 raft_consensus.cc:2229] T a2148f4eb9ba479e8ff9e4a18625df03 P 0ac449ca6f224df286ce3385bf04f72c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:24:17.649305 21957 raft_consensus.cc:2258] T a2148f4eb9ba479e8ff9e4a18625df03 P 0ac449ca6f224df286ce3385bf04f72c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:24:17.649791 21959 ts_tablet_manager.cc:1905] T a2148f4eb9ba479e8ff9e4a18625df03 P 96433db4071e4010a39d7d91f61fb62c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:24:17.649827 21957 ts_tablet_manager.cc:1905] T a2148f4eb9ba479e8ff9e4a18625df03 P 0ac449ca6f224df286ce3385bf04f72c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:24:17.652371 21957 ts_tablet_manager.cc:1918] T a2148f4eb9ba479e8ff9e4a18625df03 P 0ac449ca6f224df286ce3385bf04f72c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:24:17.652154 21959 ts_tablet_manager.cc:1918] T a2148f4eb9ba479e8ff9e4a18625df03 P 96433db4071e4010a39d7d91f61fb62c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:24:17.652460 21957 log.cc:1196] T a2148f4eb9ba479e8ff9e4a18625df03 P 0ac449ca6f224df286ce3385bf04f72c: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/ts-2/wal/wals/a2148f4eb9ba479e8ff9e4a18625df03
I20260925 15:24:17.652491 21959 log.cc:1196] T a2148f4eb9ba479e8ff9e4a18625df03 P 96433db4071e4010a39d7d91f61fb62c: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790349610595785-3919-0/minicluster-data/ts-1/wal/wals/a2148f4eb9ba479e8ff9e4a18625df03
I20260925 15:24:17.652741 21957 ts_tablet_manager.cc:1939] T a2148f4eb9ba479e8ff9e4a18625df03 P 0ac449ca6f224df286ce3385bf04f72c: Deleting consensus metadata
I20260925 15:24:17.652851 21959 ts_tablet_manager.cc:1939] T a2148f4eb9ba479e8ff9e4a18625df03 P 96433db4071e4010a39d7d91f61fb62c: Deleting consensus metadata
I20260925 15:24:17.653522 21453 catalog_manager.cc:4841] TS 96433db4071e4010a39d7d91f61fb62c (127.3.211.194:40679): tablet a2148f4eb9ba479e8ff9e4a18625df03 (table test-table [id=24fa831db78c4508a49eee3c06802ccf]) successfully deleted
I20260925 15:24:17.656255 21452 catalog_manager.cc:4841] TS 0ac449ca6f224df286ce3385bf04f72c (127.3.211.195:43331): tablet a2148f4eb9ba479e8ff9e4a18625df03 (table test-table [id=24fa831db78c4508a49eee3c06802ccf]) successfully deleted
W20260925 15:24:17.657017 21452 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.211.193:40063 (error 108)
I20260925 15:24:17.657269 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 21657
I20260925 15:24:17.657593 21925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:17.657451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:40063 (local address 127.0.0.1:50574)
0925 15:24:17.657511 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:17.657552 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.193:40063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 15:24:17.659121 21452 catalog_manager.cc:4584] TS 09e9ccf001d64dada2c228f18d45bb88 (127.3.211.193:40063): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a2148f4eb9ba479e8ff9e4a18625df03: Network error: Client connection negotiation failed: client connection to 127.3.211.193:40063: connect: Connection refused (error 111)
I20260925 15:24:17.662712 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 21788
I20260925 15:24:17.667728 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 21435
2026-09-25T15:24:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1170 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:24:17 dist-test-slave-4bk3 krb5kdc[21963](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:24:17 dist-test-slave-4bk3 krb5kdc[21963](info): set up 2 sockets
Sep 25 15:24:17 dist-test-slave-4bk3 krb5kdc[21963](info): commencing operation
krb5kdc: starting...
W20260925 15:24:18.736210 3919 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Sep 25 15:24:18 dist-test-slave-4bk3 krb5kdc[21963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:24:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:24:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.211.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
I20260925 15:24:18.868170 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:44647
--webserver_interface=127.3.211.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:33003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.211.254:44647
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:24:18.959157 21976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:18.959395 21976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:18.959422 21976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:18.960682 21976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:24:18.960728 21976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:18.960743 21976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:18.960757 21976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:24:18.960769 21976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:24:18.963680 21976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:33003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.211.254:44647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:44647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.211.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:24:18.963968 21976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:18.964167 21976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 15:24:18.969569 21976 server_base.cc:1034] running on GCE node
W20260925 15:24:18.969533 21982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:18.969534 21981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:18.969513 21984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:18.970081 21976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:18.970428 21976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:18.971632 21976 hybrid_clock.cc:648] HybridClock initialized: now 1790349858971608 us; error 49 us; skew 500 ppm
Sep 25 15:24:18 dist-test-slave-4bk3 krb5kdc[21963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349858, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:24:18.974457 21976 init.cc:377] Logged in from keytab as kudu/127.3.211.254@KRBTEST.COM (short username kudu)
I20260925 15:24:18.975036 21976 webserver.cc:460] Webserver started at http://127.3.211.254:41307/ using document root <none> and password file <none>
I20260925 15:24:18.975296 21976 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:18.975370 21976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:18.975528 21976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:18.976758 21976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/master-0/data/instance:
uuid: "7b7abceb716d46f3b7fc4a70f2136e6b"
format_stamp: "Formatted at 2026-09-25 15:24:18 on dist-test-slave-4bk3"
server_key: "c4a5f720e70741ded75e8eea2fd3bc17"
server_key_iv: "8fdec628b3bd036cac8ca29aa8f253e7"
server_key_version: "encryptionkey@0"
I20260925 15:24:18.977145 21976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/master-0/wal/instance:
uuid: "7b7abceb716d46f3b7fc4a70f2136e6b"
format_stamp: "Formatted at 2026-09-25 15:24:18 on dist-test-slave-4bk3"
server_key: "c4a5f720e70741ded75e8eea2fd3bc17"
server_key_iv: "8fdec628b3bd036cac8ca29aa8f253e7"
server_key_version: "encryptionkey@0"
I20260925 15:24:18.978619 21976 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:24:18.979789 21991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:18.980232 21976 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:24:18.980400 21976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/master-0/wal
uuid: "7b7abceb716d46f3b7fc4a70f2136e6b"
format_stamp: "Formatted at 2026-09-25 15:24:18 on dist-test-slave-4bk3"
server_key: "c4a5f720e70741ded75e8eea2fd3bc17"
server_key_iv: "8fdec628b3bd036cac8ca29aa8f253e7"
server_key_version: "encryptionkey@0"
I20260925 15:24:18.980499 21976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:18.995357 21976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:18.998600 21976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:18.998857 21976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:19.003737 21976 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.254:44647
I20260925 15:24:19.003779 22043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.254:44647 every 8 connection(s)
I20260925 15:24:19.004246 21976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
I20260925 15:24:19.005208 22044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:19.007584 22044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b7abceb716d46f3b7fc4a70f2136e6b: Bootstrap starting.
I20260925 15:24:19.008229 22044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b7abceb716d46f3b7fc4a70f2136e6b: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:19.008587 22044 log.cc:824] T 00000000000000000000000000000000 P 7b7abceb716d46f3b7fc4a70f2136e6b: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:19.009953 22044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b7abceb716d46f3b7fc4a70f2136e6b: No bootstrap required, opened a new log
I20260925 15:24:19.011650 22044 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b7abceb716d46f3b7fc4a70f2136e6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b7abceb716d46f3b7fc4a70f2136e6b" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 44647 } }
I20260925 15:24:19.011808 22044 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b7abceb716d46f3b7fc4a70f2136e6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:24:19.011831 22044 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b7abceb716d46f3b7fc4a70f2136e6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b7abceb716d46f3b7fc4a70f2136e6b, State: Initialized, Role: FOLLOWER
I20260925 15:24:19.012003 22044 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b7abceb716d46f3b7fc4a70f2136e6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b7abceb716d46f3b7fc4a70f2136e6b" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 44647 } }
I20260925 15:24:19.012084 22044 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b7abceb716d46f3b7fc4a70f2136e6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:24:19.012133 22044 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b7abceb716d46f3b7fc4a70f2136e6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:24:19.012194 22044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b7abceb716d46f3b7fc4a70f2136e6b [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:19.012878 22044 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b7abceb716d46f3b7fc4a70f2136e6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b7abceb716d46f3b7fc4a70f2136e6b" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 44647 } }
I20260925 15:24:19.012976 22044 leader_election.cc:302] T 00000000000000000000000000000000 P 7b7abceb716d46f3b7fc4a70f2136e6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b7abceb716d46f3b7fc4a70f2136e6b; no voters:
I20260925 15:24:19.013226 22044 leader_election.cc:288] T 00000000000000000000000000000000 P 7b7abceb716d46f3b7fc4a70f2136e6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:24:19.013271 22047 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b7abceb716d46f3b7fc4a70f2136e6b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:24:19.013517 22044 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b7abceb716d46f3b7fc4a70f2136e6b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:24:19.013521 22047 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b7abceb716d46f3b7fc4a70f2136e6b [term 1 LEADER]: Becoming Leader. State: Replica: 7b7abceb716d46f3b7fc4a70f2136e6b, State: Running, Role: LEADER
I20260925 15:24:19.013649 22047 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b7abceb716d46f3b7fc4a70f2136e6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b7abceb716d46f3b7fc4a70f2136e6b" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 44647 } }
I20260925 15:24:19.014211 22049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b7abceb716d46f3b7fc4a70f2136e6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b7abceb716d46f3b7fc4a70f2136e6b. Latest consensus state: current_term: 1 leader_uuid: "7b7abceb716d46f3b7fc4a70f2136e6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b7abceb716d46f3b7fc4a70f2136e6b" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 44647 } } }
I20260925 15:24:19.014302 22049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b7abceb716d46f3b7fc4a70f2136e6b [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:19.014324 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 21976
I20260925 15:24:19.014430 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/master-0/wal/instance
I20260925 15:24:19.014624 3919 external_mini_cluster.cc:1428] Setting key ee8fdd0acd2d6bf4fd74a4c005f9963d
I20260925 15:24:19.014859 22048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b7abceb716d46f3b7fc4a70f2136e6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b7abceb716d46f3b7fc4a70f2136e6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b7abceb716d46f3b7fc4a70f2136e6b" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 44647 } } }
I20260925 15:24:19.015211 22048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b7abceb716d46f3b7fc4a70f2136e6b [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:19.015043 22056 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:24:19.015767 22056 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:24:19.018256 22056 catalog_manager.cc:1347] Generated new cluster ID: 1484146e87b847a29df7b43ccfd5db05
I20260925 15:24:19.018322 22056 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 15:24:19 dist-test-slave-4bk3 krb5kdc[21963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:24:19.025542 22065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:19.015284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38088 (local address 127.3.211.254:44647)
0925 15:24:19.015471 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:19.015483 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:19.015521 (+ 38us) server_negotiation.cc:407] Connection header received
0925 15:24:19.015661 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:19.015671 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:19.015737 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:19.015901 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:19.016677 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:19.018096 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:19.019013 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:19.019473 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:19.019622 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:19.022094 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:19.022098 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:19.022098 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:19.022112 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:19.023871 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:19.024267 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:19.024270 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:19.024271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:19.024440 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:19.024660 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:19.024661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:19.024661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:19.024934 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:19.025108 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:19.025183 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 15:24:19.025346 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":75,"threads_started":1}
I20260925 15:24:19.029721 22056 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:24:19.030499 22056 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:24:19.037511 22056 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b7abceb716d46f3b7fc4a70f2136e6b: Generated new TSK 0
I20260925 15:24:19.037737 22056 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.211.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab.
Entry for principal kudu/127.3.211.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab.
I20260925 15:24:19.075234 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.193:0
--local_ip_for_outbound_sockets=127.3.211.193
--webserver_interface=127.3.211.193
--webserver_port=0
--tserver_master_addrs=127.3.211.254:44647
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab
--principal=kudu/127.3.211.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:33003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:24:19.174847 22070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:19.175107 22070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:19.175135 22070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:19.176942 22070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:19.177021 22070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:19.177062 22070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.193
I20260925 15:24:19.178946 22070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:33003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab
--principal=kudu/127.3.211.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.211.193
--webserver_port=0
--tserver_master_addrs=127.3.211.254:44647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.193
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:24:19.179217 22070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:19.179425 22070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:19.185287 22075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:19.185287 22076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:19.185325 22078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:19.186120 22070 server_base.cc:1034] running on GCE node
I20260925 15:24:19.186523 22070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:19.186930 22070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:19.188160 22070 hybrid_clock.cc:648] HybridClock initialized: now 1790349859188101 us; error 72 us; skew 500 ppm
Sep 25 15:24:19 dist-test-slave-4bk3 krb5kdc[21963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349859, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:24:19.191133 22070 init.cc:377] Logged in from keytab as kudu/127.3.211.193@KRBTEST.COM (short username kudu)
I20260925 15:24:19.191732 22070 webserver.cc:460] Webserver started at http://127.3.211.193:34365/ using document root <none> and password file <none>
I20260925 15:24:19.192000 22070 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:19.192075 22070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:19.192229 22070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:19.193411 22070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/ts-0/data/instance:
uuid: "864e26e1ad044f80be9962f1b030fad7"
format_stamp: "Formatted at 2026-09-25 15:24:19 on dist-test-slave-4bk3"
server_key: "5032754ac24932495d621769820d4539"
server_key_iv: "35c6453486348c79ee0806f64e149b14"
server_key_version: "encryptionkey@0"
I20260925 15:24:19.193791 22070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/ts-0/wal/instance:
uuid: "864e26e1ad044f80be9962f1b030fad7"
format_stamp: "Formatted at 2026-09-25 15:24:19 on dist-test-slave-4bk3"
server_key: "5032754ac24932495d621769820d4539"
server_key_iv: "35c6453486348c79ee0806f64e149b14"
server_key_version: "encryptionkey@0"
I20260925 15:24:19.195278 22070 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:24:19.196575 22085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:19.196859 22070 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:24:19.196985 22070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/ts-0/wal
uuid: "864e26e1ad044f80be9962f1b030fad7"
format_stamp: "Formatted at 2026-09-25 15:24:19 on dist-test-slave-4bk3"
server_key: "5032754ac24932495d621769820d4539"
server_key_iv: "35c6453486348c79ee0806f64e149b14"
server_key_version: "encryptionkey@0"
I20260925 15:24:19.197079 22070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:19.208477 22070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:19.211193 22070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:19.211387 22070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:19.211654 22070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:24:19.212010 22070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:24:19.212067 22070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:19.212106 22070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:24:19.212132 22070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:19.219010 22070 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.193:35287
I20260925 15:24:19.219105 22198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.193:35287 every 8 connection(s)
I20260925 15:24:19.219462 22070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:24:19.221557 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 22070
I20260925 15:24:19.221675 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/ts-0/wal/instance
I20260925 15:24:19.221896 3919 external_mini_cluster.cc:1428] Setting key 7a185f60e863186377483d43a8276f13
Sep 25 15:24:19 dist-test-slave-4bk3 krb5kdc[21963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349859, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.193@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.211.194@KRBTEST.COM; defaulting to no policy
I20260925 15:24:19.233407 22065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:19.220572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:43087 (local address 127.3.211.254:44647)
0925 15:24:19.220620 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:19.220623 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:19.220994 (+ 371us) server_negotiation.cc:407] Connection header received
0925 15:24:19.221494 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:19.221496 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:19.221517 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:19.221606 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:19.222815 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:19.223848 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:19.226114 (+ 2266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:19.226421 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:19.226517 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:19.229118 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:19.229120 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:19.229121 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:19.229140 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:19.230965 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:19.231892 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:19.231894 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:19.231894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:19.231965 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:19.232485 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:19.232486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:19.232487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:19.232660 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:19.232725 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:19.233168 (+ 443us) server_negotiation.cc:299] Negotiation successful
0925 15:24:19.233327 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 15:24:19.233668 22201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:19.220586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:44647 (local address 127.3.211.193:43087)
0925 15:24:19.220816 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:19.220861 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:19.221377 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:19.221707 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:19.221712 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:19.222483 (+ 771us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:24:19.222700 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:19.222705 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:19.224041 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:19.224043 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:19.225918 (+ 1875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:19.225924 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:19.226597 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:19.226599 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:19.226724 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:19.227396 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:24:19.227406 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:24:19.228981 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:24:19.231138 (+ 2157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:19.231149 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:19.231152 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:19.231741 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:19.232104 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:19.232106 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:19.232106 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:19.232392 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:19.232769 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:24:19.232778 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:24:19.232945 (+ 167us) client_negotiation.cc:769] Sending connection context
0925 15:24:19.233069 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 15:24:19.233340 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":122,"threads_started":1}
I20260925 15:24:19.234442 22199 heartbeater.cc:347] Connected to a master server at 127.3.211.254:44647
I20260925 15:24:19.234549 22199 heartbeater.cc:464] Registering TS with master...
I20260925 15:24:19.234715 22199 heartbeater.cc:511] Master 127.3.211.254:44647 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.194@KRBTEST.COM" created.
I20260925 15:24:19.235404 22008 ts_manager.cc:194] Registered new tserver with Master: 864e26e1ad044f80be9962f1b030fad7 (127.3.211.193:35287)
I20260925 15:24:19.236075 22008 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.193@KRBTEST.COM'} at 127.3.211.193:43087
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab.
Entry for principal kudu/127.3.211.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab.
I20260925 15:24:19.252548 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.194:0
--local_ip_for_outbound_sockets=127.3.211.194
--webserver_interface=127.3.211.194
--webserver_port=0
--tserver_master_addrs=127.3.211.254:44647
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab
--principal=kudu/127.3.211.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:33003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:24:19.342479 22204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:19.342712 22204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:19.342741 22204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:19.343958 22204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:19.344004 22204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:19.344033 22204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.194
I20260925 15:24:19.345677 22204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:33003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab
--principal=kudu/127.3.211.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.211.194
--webserver_port=0
--tserver_master_addrs=127.3.211.254:44647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.194
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:24:19.345913 22204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:19.346107 22204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:19.351667 22209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:19.351739 22210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:19.352011 22204 server_base.cc:1034] running on GCE node
W20260925 15:24:19.351739 22212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:19.352490 22204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:19.352798 22204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:19.354035 22204 hybrid_clock.cc:648] HybridClock initialized: now 1790349859354003 us; error 57 us; skew 500 ppm
Sep 25 15:24:19 dist-test-slave-4bk3 krb5kdc[21963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349859, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:24:19.357908 22204 init.cc:377] Logged in from keytab as kudu/127.3.211.194@KRBTEST.COM (short username kudu)
I20260925 15:24:19.358858 22204 webserver.cc:460] Webserver started at http://127.3.211.194:41989/ using document root <none> and password file <none>
I20260925 15:24:19.359119 22204 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:19.359195 22204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:19.359442 22204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:19.360859 22204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/ts-1/data/instance:
uuid: "eda9ed42311940eba66ed22f4af810a4"
format_stamp: "Formatted at 2026-09-25 15:24:19 on dist-test-slave-4bk3"
server_key: "0d9f294a468bc31d267b4ec623ad1e83"
server_key_iv: "e36e5811611725d504b031735c0b0508"
server_key_version: "encryptionkey@0"
I20260925 15:24:19.361251 22204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/ts-1/wal/instance:
uuid: "eda9ed42311940eba66ed22f4af810a4"
format_stamp: "Formatted at 2026-09-25 15:24:19 on dist-test-slave-4bk3"
server_key: "0d9f294a468bc31d267b4ec623ad1e83"
server_key_iv: "e36e5811611725d504b031735c0b0508"
server_key_version: "encryptionkey@0"
I20260925 15:24:19.363071 22204 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:24:19.364390 22219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:19.364626 22204 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:24:19.364784 22204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/ts-1/wal
uuid: "eda9ed42311940eba66ed22f4af810a4"
format_stamp: "Formatted at 2026-09-25 15:24:19 on dist-test-slave-4bk3"
server_key: "0d9f294a468bc31d267b4ec623ad1e83"
server_key_iv: "e36e5811611725d504b031735c0b0508"
server_key_version: "encryptionkey@0"
I20260925 15:24:19.364887 22204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:19.374799 22204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:19.377463 22204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:19.377790 22204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:19.378106 22204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:24:19.378471 22204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:24:19.378535 22204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:19.378580 22204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:24:19.378613 22204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:19.387244 22204 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.194:39703
I20260925 15:24:19.387398 22332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.194:39703 every 8 connection(s)
I20260925 15:24:19.387692 22204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
I20260925 15:24:19.388290 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 22204
I20260925 15:24:19.388445 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/ts-1/wal/instance
I20260925 15:24:19.388708 3919 external_mini_cluster.cc:1428] Setting key 27b503606ca1e9370c5164ec098734a9
WARNING: no policy specified for kudu/127.3.211.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.195@KRBTEST.COM" created.
Sep 25 15:24:19 dist-test-slave-4bk3 krb5kdc[21963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349859, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.194@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:24:19.408525 22065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:19.388547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:40035 (local address 127.3.211.254:44647)
0925 15:24:19.388604 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:19.388606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:19.388943 (+ 337us) server_negotiation.cc:407] Connection header received
0925 15:24:19.389548 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:19.389549 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:19.389558 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:19.389594 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:19.391025 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:19.391877 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:19.393161 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:19.393493 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:19.393679 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:19.404211 (+ 10532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:19.404213 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:19.404214 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:19.404225 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:19.406442 (+ 2217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:19.407185 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:19.407186 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:19.407187 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:19.407270 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:19.407698 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:19.407700 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:19.407700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:19.407879 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:19.407973 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:19.408278 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 15:24:19.408408 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 15:24:19.408756 22335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:19.388554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:44647 (local address 127.3.211.194:40035)
0925 15:24:19.388792 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:19.388847 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:19.389426 (+ 579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:19.389682 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:19.389687 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:19.390549 (+ 862us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:24:19.390844 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:19.390851 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:19.392089 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:19.392091 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:19.393053 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:19.393056 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:19.393700 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:19.393701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:19.393852 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:19.402371 (+ 8519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:24:19.402386 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:24:19.404067 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:24:19.406610 (+ 2543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:19.406630 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:19.406634 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:19.407055 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:19.407432 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:19.407434 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:19.407435 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:19.407607 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:19.407967 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:24:19.407974 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:24:19.408126 (+ 152us) client_negotiation.cc:769] Sending connection context
0925 15:24:19.408229 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 15:24:19.408484 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":88,"threads_started":1}
I20260925 15:24:19.409336 22333 heartbeater.cc:347] Connected to a master server at 127.3.211.254:44647
I20260925 15:24:19.409451 22333 heartbeater.cc:464] Registering TS with master...
I20260925 15:24:19.409662 22333 heartbeater.cc:511] Master 127.3.211.254:44647 requested a full tablet report, sending...
I20260925 15:24:19.410106 22008 ts_manager.cc:194] Registered new tserver with Master: eda9ed42311940eba66ed22f4af810a4 (127.3.211.194:39703)
I20260925 15:24:19.410645 22008 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.194@KRBTEST.COM'} at 127.3.211.194:40035
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab.
Entry for principal kudu/127.3.211.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab.
I20260925 15:24:19.421001 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.195:0
--local_ip_for_outbound_sockets=127.3.211.195
--webserver_interface=127.3.211.195
--webserver_port=0
--tserver_master_addrs=127.3.211.254:44647
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab
--principal=kudu/127.3.211.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:33003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:24:19.514532 22338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:19.514832 22338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:19.514882 22338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:19.516242 22338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:19.516299 22338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:19.516384 22338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.195
I20260925 15:24:19.518105 22338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:33003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab
--principal=kudu/127.3.211.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.211.195
--webserver_port=0
--tserver_master_addrs=127.3.211.254:44647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.195
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:24:19.518328 22338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:19.518626 22338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:19.524013 22346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:19.524024 22343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:19.524029 22338 server_base.cc:1034] running on GCE node
W20260925 15:24:19.524224 22344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:19.524507 22338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:19.524787 22338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:19.525975 22338 hybrid_clock.cc:648] HybridClock initialized: now 1790349859525933 us; error 57 us; skew 500 ppm
Sep 25 15:24:19 dist-test-slave-4bk3 krb5kdc[21963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349859, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:24:19.528916 22338 init.cc:377] Logged in from keytab as kudu/127.3.211.195@KRBTEST.COM (short username kudu)
I20260925 15:24:19.529464 22338 webserver.cc:460] Webserver started at http://127.3.211.195:45819/ using document root <none> and password file <none>
I20260925 15:24:19.529701 22338 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:19.529770 22338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:19.529929 22338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:19.531076 22338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/ts-2/data/instance:
uuid: "a46aa5ae2fe844b79c9e5999697b4469"
format_stamp: "Formatted at 2026-09-25 15:24:19 on dist-test-slave-4bk3"
server_key: "b9bb5f6f8566dc6907ce6c33b6073f7a"
server_key_iv: "567c0b15bd0e54bd2dfe89fd66af7a8f"
server_key_version: "encryptionkey@0"
I20260925 15:24:19.531482 22338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/ts-2/wal/instance:
uuid: "a46aa5ae2fe844b79c9e5999697b4469"
format_stamp: "Formatted at 2026-09-25 15:24:19 on dist-test-slave-4bk3"
server_key: "b9bb5f6f8566dc6907ce6c33b6073f7a"
server_key_iv: "567c0b15bd0e54bd2dfe89fd66af7a8f"
server_key_version: "encryptionkey@0"
I20260925 15:24:19.533136 22338 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:24:19.534099 22353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:19.534453 22338 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:24:19.534593 22338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/ts-2/wal
uuid: "a46aa5ae2fe844b79c9e5999697b4469"
format_stamp: "Formatted at 2026-09-25 15:24:19 on dist-test-slave-4bk3"
server_key: "b9bb5f6f8566dc6907ce6c33b6073f7a"
server_key_iv: "567c0b15bd0e54bd2dfe89fd66af7a8f"
server_key_version: "encryptionkey@0"
I20260925 15:24:19.534687 22338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:19.543677 22338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:19.546437 22338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:19.546623 22338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:19.546890 22338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:24:19.547219 22338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:24:19.547277 22338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:19.547324 22338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:24:19.547358 22338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:19.554572 22338 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.195:34023
I20260925 15:24:19.554711 22466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.195:34023 every 8 connection(s)
I20260925 15:24:19.555125 22338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
I20260925 15:24:19.557932 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 22338
I20260925 15:24:19.558100 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/ts-2/wal/instance
I20260925 15:24:19.558338 3919 external_mini_cluster.cc:1428] Setting key 93917545af4cf6432de446199c2d1550
Sep 25 15:24:19 dist-test-slave-4bk3 krb5kdc[21963](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349859, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.195@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:24:19.569011 22065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:19.556127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:37737 (local address 127.3.211.254:44647)
0925 15:24:19.556207 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:19.556210 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:19.556647 (+ 437us) server_negotiation.cc:407] Connection header received
0925 15:24:19.557188 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:19.557189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:19.557201 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:19.557266 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:19.559497 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:19.560315 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:19.561833 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:19.562052 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:19.562140 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:19.565381 (+ 3241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:19.565383 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:19.565384 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:19.565394 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:19.566993 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:19.567632 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:19.567633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:19.567633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:19.567693 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:19.568138 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:19.568140 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:19.568140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:19.568300 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:19.568461 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:19.568806 (+ 345us) server_negotiation.cc:299] Negotiation successful
0925 15:24:19.568902 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 15:24:19.569233 22469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:19.556102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:44647 (local address 127.3.211.195:37737)
0925 15:24:19.556480 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:19.556531 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:19.557038 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:19.557487 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:19.557494 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:19.558677 (+ 1183us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:24:19.559278 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:19.559299 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:19.560516 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:19.560517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:19.561657 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:19.561661 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:19.562242 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:19.562244 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:19.562435 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:19.563079 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:24:19.563092 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:24:19.565181 (+ 2089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:24:19.567181 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:19.567195 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:19.567198 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:19.567522 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:19.567818 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:19.567819 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:19.567820 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:19.568047 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:19.568454 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:24:19.568462 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:24:19.568655 (+ 193us) client_negotiation.cc:769] Sending connection context
0925 15:24:19.568781 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 15:24:19.568991 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":124,"threads_started":1}
I20260925 15:24:19.570442 22467 heartbeater.cc:347] Connected to a master server at 127.3.211.254:44647
I20260925 15:24:19.570566 22467 heartbeater.cc:464] Registering TS with master...
I20260925 15:24:19.570806 22467 heartbeater.cc:511] Master 127.3.211.254:44647 requested a full tablet report, sending...
I20260925 15:24:19.571239 22008 ts_manager.cc:194] Registered new tserver with Master: a46aa5ae2fe844b79c9e5999697b4469 (127.3.211.195:34023)
I20260925 15:24:19.571655 22008 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.195@KRBTEST.COM'} at 127.3.211.195:37737
I20260925 15:24:19.583026 3919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:24:19.592278 22065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:19.584615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38098 (local address 127.3.211.254:44647)
0925 15:24:19.584712 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:19.584715 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:19.584806 (+ 91us) server_negotiation.cc:407] Connection header received
0925 15:24:19.584928 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:19.584929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:19.584941 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:19.584998 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:19.585611 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:19.586119 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:19.586847 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:19.587047 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:19.587123 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:19.588646 (+ 1523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:19.588648 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:19.588648 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:19.588657 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:19.590369 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:19.591016 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:19.591018 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:19.591018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:19.591106 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:19.591346 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:19.591348 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:19.591348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:19.591527 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:19.591842 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:19.592044 (+ 202us) server_negotiation.cc:299] Negotiation successful
0925 15:24:19.592136 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 15:24:19.595055 22008 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38098:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:24:19.596927 22008 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:24:19.604305 22478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:19.599724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.195:34023 (local address 127.0.0.1:32836)
0925 15:24:19.600146 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:19.600191 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:19.600304 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:19.600577 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:19.600580 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:19.600583 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:24:19.600673 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:19.600675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:19.601796 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:19.601797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:19.603178 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:19.603179 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:19.603952 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:19.603953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:19.604049 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:19.604064 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 15:24:19.604140 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 15:24:19.604204 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"mutex_wait_us":1,"thread_start_us":68,"threads_started":1}
I20260925 15:24:19.604502 22479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:19.599868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:35287 (local address 127.0.0.1:43972)
0925 15:24:19.600299 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:19.600309 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:19.600387 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:19.600659 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:19.600660 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:19.600662 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:24:19.600711 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:19.600713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:19.602081 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:19.602082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:19.603175 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:19.603178 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:19.604216 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:19.604217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:19.604291 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:19.604295 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:24:19.604364 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 15:24:19.604410 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"mutex_wait_us":4,"thread_start_us":106,"threads_started":1}
I20260925 15:24:19.604422 22482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:19.600054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54550 (local address 127.3.211.194:39703)
0925 15:24:19.600222 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:19.600227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:19.600909 (+ 682us) server_negotiation.cc:407] Connection header received
0925 15:24:19.600953 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:19.600956 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:19.600971 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:19.601052 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:24:19.601170 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:19.602189 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:19.603347 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:19.604055 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:19.604124 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:19.604179 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:19.604264 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 15:24:19.604313 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":102,"threads_started":1}
I20260925 15:24:19.604803 22483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:19.600130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43972 (local address 127.3.211.193:35287)
0925 15:24:19.600289 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:19.600295 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:19.600464 (+ 169us) server_negotiation.cc:407] Connection header received
0925 15:24:19.600533 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:19.600536 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:19.600552 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:19.600624 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:24:19.600796 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:19.601974 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:19.603289 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:19.604046 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:19.604123 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:19.604179 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:19.604680 (+ 501us) server_negotiation.cc:299] Negotiation successful
0925 15:24:19.604734 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":85,"threads_started":1}
I20260925 15:24:19.605046 22480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:19.599888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32836 (local address 127.3.211.195:34023)
0925 15:24:19.600082 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:19.600087 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:19.600259 (+ 172us) server_negotiation.cc:407] Connection header received
0925 15:24:19.600464 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:19.600468 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:19.600487 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:19.600559 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:24:19.600756 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:19.601684 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:19.603286 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:19.603850 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:19.603951 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:19.604017 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:19.604777 (+ 760us) server_negotiation.cc:299] Negotiation successful
0925 15:24:19.604866 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":85,"threads_started":1}
I20260925 15:24:19.605360 22481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:19.599959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.194:39703 (local address 127.0.0.1:54550)
0925 15:24:19.600721 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:19.600730 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:19.600777 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:19.601065 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:19.601066 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:19.601069 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:24:19.601107 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:19.601108 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:19.602298 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:19.602299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:19.603260 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:19.603264 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:19.604141 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:19.604142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:19.604196 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:19.604199 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:24:19.604250 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 15:24:19.604285 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"thread_start_us":88,"threads_started":1}
I20260925 15:24:19.605643 22401 tablet_service.cc:1467] Processing CreateTablet for tablet 8627689917d84f13a5d2f06127bc47e0 (DEFAULT_TABLE table=test-table [id=81a2e154c1cb47e49d42ea04e9719021]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:24:19.605643 22133 tablet_service.cc:1467] Processing CreateTablet for tablet 8627689917d84f13a5d2f06127bc47e0 (DEFAULT_TABLE table=test-table [id=81a2e154c1cb47e49d42ea04e9719021]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:24:19.606029 22133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8627689917d84f13a5d2f06127bc47e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:19.606029 22401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8627689917d84f13a5d2f06127bc47e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:19.608945 22267 tablet_service.cc:1467] Processing CreateTablet for tablet 8627689917d84f13a5d2f06127bc47e0 (DEFAULT_TABLE table=test-table [id=81a2e154c1cb47e49d42ea04e9719021]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:24:19.609328 22267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8627689917d84f13a5d2f06127bc47e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:19.609733 22484 tablet_bootstrap.cc:492] T 8627689917d84f13a5d2f06127bc47e0 P a46aa5ae2fe844b79c9e5999697b4469: Bootstrap starting.
I20260925 15:24:19.610754 22484 tablet_bootstrap.cc:654] T 8627689917d84f13a5d2f06127bc47e0 P a46aa5ae2fe844b79c9e5999697b4469: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:19.611786 22484 log.cc:824] T 8627689917d84f13a5d2f06127bc47e0 P a46aa5ae2fe844b79c9e5999697b4469: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:19.612566 22485 tablet_bootstrap.cc:492] T 8627689917d84f13a5d2f06127bc47e0 P 864e26e1ad044f80be9962f1b030fad7: Bootstrap starting.
I20260925 15:24:19.612929 22484 tablet_bootstrap.cc:492] T 8627689917d84f13a5d2f06127bc47e0 P a46aa5ae2fe844b79c9e5999697b4469: No bootstrap required, opened a new log
I20260925 15:24:19.613029 22484 ts_tablet_manager.cc:1397] T 8627689917d84f13a5d2f06127bc47e0 P a46aa5ae2fe844b79c9e5999697b4469: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 15:24:19.613315 22487 tablet_bootstrap.cc:492] T 8627689917d84f13a5d2f06127bc47e0 P eda9ed42311940eba66ed22f4af810a4: Bootstrap starting.
I20260925 15:24:19.613385 22485 tablet_bootstrap.cc:654] T 8627689917d84f13a5d2f06127bc47e0 P 864e26e1ad044f80be9962f1b030fad7: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:19.613976 22485 log.cc:824] T 8627689917d84f13a5d2f06127bc47e0 P 864e26e1ad044f80be9962f1b030fad7: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:19.614137 22487 tablet_bootstrap.cc:654] T 8627689917d84f13a5d2f06127bc47e0 P eda9ed42311940eba66ed22f4af810a4: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:19.614670 22487 log.cc:824] T 8627689917d84f13a5d2f06127bc47e0 P eda9ed42311940eba66ed22f4af810a4: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:19.614881 22485 tablet_bootstrap.cc:492] T 8627689917d84f13a5d2f06127bc47e0 P 864e26e1ad044f80be9962f1b030fad7: No bootstrap required, opened a new log
I20260925 15:24:19.614981 22485 ts_tablet_manager.cc:1397] T 8627689917d84f13a5d2f06127bc47e0 P 864e26e1ad044f80be9962f1b030fad7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:24:19.615486 22487 tablet_bootstrap.cc:492] T 8627689917d84f13a5d2f06127bc47e0 P eda9ed42311940eba66ed22f4af810a4: No bootstrap required, opened a new log
I20260925 15:24:19.615567 22487 ts_tablet_manager.cc:1397] T 8627689917d84f13a5d2f06127bc47e0 P eda9ed42311940eba66ed22f4af810a4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:24:19.615522 22484 raft_consensus.cc:348] T 8627689917d84f13a5d2f06127bc47e0 P a46aa5ae2fe844b79c9e5999697b4469 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aa5ae2fe844b79c9e5999697b4469" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 34023 } } peers { permanent_uuid: "eda9ed42311940eba66ed22f4af810a4" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 39703 } } peers { permanent_uuid: "864e26e1ad044f80be9962f1b030fad7" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 35287 } }
I20260925 15:24:19.615729 22484 raft_consensus.cc:374] T 8627689917d84f13a5d2f06127bc47e0 P a46aa5ae2fe844b79c9e5999697b4469 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:24:19.615772 22484 raft_consensus.cc:729] T 8627689917d84f13a5d2f06127bc47e0 P a46aa5ae2fe844b79c9e5999697b4469 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a46aa5ae2fe844b79c9e5999697b4469, State: Initialized, Role: FOLLOWER
I20260925 15:24:19.615919 22484 consensus_queue.cc:260] T 8627689917d84f13a5d2f06127bc47e0 P a46aa5ae2fe844b79c9e5999697b4469 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aa5ae2fe844b79c9e5999697b4469" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 34023 } } peers { permanent_uuid: "eda9ed42311940eba66ed22f4af810a4" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 39703 } } peers { permanent_uuid: "864e26e1ad044f80be9962f1b030fad7" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 35287 } }
I20260925 15:24:19.616216 22484 ts_tablet_manager.cc:1428] T 8627689917d84f13a5d2f06127bc47e0 P a46aa5ae2fe844b79c9e5999697b4469: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:24:19.616369 22467 heartbeater.cc:503] Master 127.3.211.254:44647 was elected leader, sending a full tablet report...
I20260925 15:24:19.617591 22487 raft_consensus.cc:348] T 8627689917d84f13a5d2f06127bc47e0 P eda9ed42311940eba66ed22f4af810a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aa5ae2fe844b79c9e5999697b4469" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 34023 } } peers { permanent_uuid: "eda9ed42311940eba66ed22f4af810a4" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 39703 } } peers { permanent_uuid: "864e26e1ad044f80be9962f1b030fad7" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 35287 } }
I20260925 15:24:19.617755 22487 raft_consensus.cc:374] T 8627689917d84f13a5d2f06127bc47e0 P eda9ed42311940eba66ed22f4af810a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:24:19.617624 22485 raft_consensus.cc:348] T 8627689917d84f13a5d2f06127bc47e0 P 864e26e1ad044f80be9962f1b030fad7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aa5ae2fe844b79c9e5999697b4469" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 34023 } } peers { permanent_uuid: "eda9ed42311940eba66ed22f4af810a4" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 39703 } } peers { permanent_uuid: "864e26e1ad044f80be9962f1b030fad7" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 35287 } }
I20260925 15:24:19.617779 22487 raft_consensus.cc:729] T 8627689917d84f13a5d2f06127bc47e0 P eda9ed42311940eba66ed22f4af810a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eda9ed42311940eba66ed22f4af810a4, State: Initialized, Role: FOLLOWER
I20260925 15:24:19.617787 22485 raft_consensus.cc:374] T 8627689917d84f13a5d2f06127bc47e0 P 864e26e1ad044f80be9962f1b030fad7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:24:19.617831 22485 raft_consensus.cc:729] T 8627689917d84f13a5d2f06127bc47e0 P 864e26e1ad044f80be9962f1b030fad7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 864e26e1ad044f80be9962f1b030fad7, State: Initialized, Role: FOLLOWER
I20260925 15:24:19.617924 22487 consensus_queue.cc:260] T 8627689917d84f13a5d2f06127bc47e0 P eda9ed42311940eba66ed22f4af810a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aa5ae2fe844b79c9e5999697b4469" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 34023 } } peers { permanent_uuid: "eda9ed42311940eba66ed22f4af810a4" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 39703 } } peers { permanent_uuid: "864e26e1ad044f80be9962f1b030fad7" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 35287 } }
I20260925 15:24:19.617962 22485 consensus_queue.cc:260] T 8627689917d84f13a5d2f06127bc47e0 P 864e26e1ad044f80be9962f1b030fad7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aa5ae2fe844b79c9e5999697b4469" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 34023 } } peers { permanent_uuid: "eda9ed42311940eba66ed22f4af810a4" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 39703 } } peers { permanent_uuid: "864e26e1ad044f80be9962f1b030fad7" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 35287 } }
I20260925 15:24:19.618290 22485 ts_tablet_manager.cc:1428] T 8627689917d84f13a5d2f06127bc47e0 P 864e26e1ad044f80be9962f1b030fad7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:24:19.618319 22199 heartbeater.cc:503] Master 127.3.211.254:44647 was elected leader, sending a full tablet report...
I20260925 15:24:19.618265 22487 ts_tablet_manager.cc:1428] T 8627689917d84f13a5d2f06127bc47e0 P eda9ed42311940eba66ed22f4af810a4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:24:19.618278 22333 heartbeater.cc:503] Master 127.3.211.254:44647 was elected leader, sending a full tablet report...
W20260925 15:24:19.638465 22334 tablet.cc:2369] T 8627689917d84f13a5d2f06127bc47e0 P eda9ed42311940eba66ed22f4af810a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:24:19.720304 22200 tablet.cc:2369] T 8627689917d84f13a5d2f06127bc47e0 P 864e26e1ad044f80be9962f1b030fad7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:24:19.805966 22468 tablet.cc:2369] T 8627689917d84f13a5d2f06127bc47e0 P a46aa5ae2fe844b79c9e5999697b4469: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:24:19.864437 22492 raft_consensus.cc:482] T 8627689917d84f13a5d2f06127bc47e0 P 864e26e1ad044f80be9962f1b030fad7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:24:19.864629 22492 raft_consensus.cc:504] T 8627689917d84f13a5d2f06127bc47e0 P 864e26e1ad044f80be9962f1b030fad7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aa5ae2fe844b79c9e5999697b4469" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 34023 } } peers { permanent_uuid: "eda9ed42311940eba66ed22f4af810a4" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 39703 } } peers { permanent_uuid: "864e26e1ad044f80be9962f1b030fad7" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 35287 } }
I20260925 15:24:19.865074 22492 leader_election.cc:288] T 8627689917d84f13a5d2f06127bc47e0 P 864e26e1ad044f80be9962f1b030fad7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a46aa5ae2fe844b79c9e5999697b4469 (127.3.211.195:34023), eda9ed42311940eba66ed22f4af810a4 (127.3.211.194:39703)
I20260925 15:24:19.870028 22480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:19.865643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:41991 (local address 127.3.211.195:34023)
0925 15:24:19.865717 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:19.865720 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:19.865945 (+ 225us) server_negotiation.cc:407] Connection header received
0925 15:24:19.865991 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:19.865992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:19.866004 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:19.866091 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:24:19.866357 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:19.867140 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:19.868953 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:19.869616 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:19.869683 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:19.869690 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:19.869902 (+ 212us) server_negotiation.cc:299] Negotiation successful
0925 15:24:19.869951 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 15:24:19.870028 22482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:19.866076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:50937 (local address 127.3.211.194:39703)
0925 15:24:19.866153 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:19.866155 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:19.866167 (+ 12us) server_negotiation.cc:407] Connection header received
0925 15:24:19.866209 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:19.866209 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:19.866216 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:19.866257 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:24:19.866428 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:19.867139 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:19.868954 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:19.869561 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:19.869658 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:19.869671 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:19.869902 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 15:24:19.869951 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 15:24:19.870054 22493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:19.865485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.195:34023 (local address 127.3.211.193:41991)
0925 15:24:19.865675 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:19.865692 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:19.865919 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:19.866153 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:19.866154 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:19.866156 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:24:19.866273 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:19.866276 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:19.867270 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:19.867271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:19.868800 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:19.868805 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:19.869703 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:19.869703 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:19.869776 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:19.869782 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 15:24:19.869860 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 15:24:19.869921 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":104,"thread_start_us":111,"threads_started":1}
I20260925 15:24:19.870054 22494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:19.865946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.194:39703 (local address 127.3.211.193:50937)
0925 15:24:19.866089 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:19.866103 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:19.866153 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:19.866285 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:19.866286 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:19.866288 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:24:19.866357 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:19.866359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:19.867267 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:19.867269 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:19.868800 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:19.868805 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:19.869685 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:19.869686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:19.869776 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:19.869784 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 15:24:19.869867 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 15:24:19.869920 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":72,"thread_start_us":70,"threads_started":1}
I20260925 15:24:19.870503 22421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8627689917d84f13a5d2f06127bc47e0" candidate_uuid: "864e26e1ad044f80be9962f1b030fad7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a46aa5ae2fe844b79c9e5999697b4469" is_pre_election: true
I20260925 15:24:19.870503 22287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8627689917d84f13a5d2f06127bc47e0" candidate_uuid: "864e26e1ad044f80be9962f1b030fad7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eda9ed42311940eba66ed22f4af810a4" is_pre_election: true
I20260925 15:24:19.870692 22421 raft_consensus.cc:2454] T 8627689917d84f13a5d2f06127bc47e0 P a46aa5ae2fe844b79c9e5999697b4469 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 864e26e1ad044f80be9962f1b030fad7 in term 0.
I20260925 15:24:19.870698 22287 raft_consensus.cc:2454] T 8627689917d84f13a5d2f06127bc47e0 P eda9ed42311940eba66ed22f4af810a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 864e26e1ad044f80be9962f1b030fad7 in term 0.
I20260925 15:24:19.870940 22086 leader_election.cc:302] T 8627689917d84f13a5d2f06127bc47e0 P 864e26e1ad044f80be9962f1b030fad7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 864e26e1ad044f80be9962f1b030fad7, a46aa5ae2fe844b79c9e5999697b4469; no voters:
I20260925 15:24:19.871088 22492 raft_consensus.cc:2781] T 8627689917d84f13a5d2f06127bc47e0 P 864e26e1ad044f80be9962f1b030fad7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:24:19.871150 22492 raft_consensus.cc:482] T 8627689917d84f13a5d2f06127bc47e0 P 864e26e1ad044f80be9962f1b030fad7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:24:19.871183 22492 raft_consensus.cc:3037] T 8627689917d84f13a5d2f06127bc47e0 P 864e26e1ad044f80be9962f1b030fad7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:19.871954 22492 raft_consensus.cc:504] T 8627689917d84f13a5d2f06127bc47e0 P 864e26e1ad044f80be9962f1b030fad7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aa5ae2fe844b79c9e5999697b4469" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 34023 } } peers { permanent_uuid: "eda9ed42311940eba66ed22f4af810a4" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 39703 } } peers { permanent_uuid: "864e26e1ad044f80be9962f1b030fad7" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 35287 } }
I20260925 15:24:19.872115 22492 leader_election.cc:288] T 8627689917d84f13a5d2f06127bc47e0 P 864e26e1ad044f80be9962f1b030fad7 [CANDIDATE]: Term 1 election: Requested vote from peers a46aa5ae2fe844b79c9e5999697b4469 (127.3.211.195:34023), eda9ed42311940eba66ed22f4af810a4 (127.3.211.194:39703)
I20260925 15:24:19.872377 22287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8627689917d84f13a5d2f06127bc47e0" candidate_uuid: "864e26e1ad044f80be9962f1b030fad7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eda9ed42311940eba66ed22f4af810a4"
I20260925 15:24:19.872318 22421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8627689917d84f13a5d2f06127bc47e0" candidate_uuid: "864e26e1ad044f80be9962f1b030fad7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a46aa5ae2fe844b79c9e5999697b4469"
I20260925 15:24:19.872478 22421 raft_consensus.cc:3037] T 8627689917d84f13a5d2f06127bc47e0 P a46aa5ae2fe844b79c9e5999697b4469 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:19.872480 22287 raft_consensus.cc:3037] T 8627689917d84f13a5d2f06127bc47e0 P eda9ed42311940eba66ed22f4af810a4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:19.873245 22287 raft_consensus.cc:2454] T 8627689917d84f13a5d2f06127bc47e0 P eda9ed42311940eba66ed22f4af810a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 864e26e1ad044f80be9962f1b030fad7 in term 1.
I20260925 15:24:19.873252 22421 raft_consensus.cc:2454] T 8627689917d84f13a5d2f06127bc47e0 P a46aa5ae2fe844b79c9e5999697b4469 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 864e26e1ad044f80be9962f1b030fad7 in term 1.
I20260925 15:24:19.873435 22086 leader_election.cc:302] T 8627689917d84f13a5d2f06127bc47e0 P 864e26e1ad044f80be9962f1b030fad7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 864e26e1ad044f80be9962f1b030fad7, a46aa5ae2fe844b79c9e5999697b4469; no voters:
I20260925 15:24:19.873565 22492 raft_consensus.cc:2781] T 8627689917d84f13a5d2f06127bc47e0 P 864e26e1ad044f80be9962f1b030fad7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:24:19.873765 22492 raft_consensus.cc:686] T 8627689917d84f13a5d2f06127bc47e0 P 864e26e1ad044f80be9962f1b030fad7 [term 1 LEADER]: Becoming Leader. State: Replica: 864e26e1ad044f80be9962f1b030fad7, State: Running, Role: LEADER
I20260925 15:24:19.873896 22492 consensus_queue.cc:237] T 8627689917d84f13a5d2f06127bc47e0 P 864e26e1ad044f80be9962f1b030fad7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aa5ae2fe844b79c9e5999697b4469" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 34023 } } peers { permanent_uuid: "eda9ed42311940eba66ed22f4af810a4" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 39703 } } peers { permanent_uuid: "864e26e1ad044f80be9962f1b030fad7" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 35287 } }
I20260925 15:24:19.875079 22007 catalog_manager.cc:5495] T 8627689917d84f13a5d2f06127bc47e0 P 864e26e1ad044f80be9962f1b030fad7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 864e26e1ad044f80be9962f1b030fad7 (127.3.211.193). New cstate: current_term: 1 leader_uuid: "864e26e1ad044f80be9962f1b030fad7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a46aa5ae2fe844b79c9e5999697b4469" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 34023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eda9ed42311940eba66ed22f4af810a4" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 39703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "864e26e1ad044f80be9962f1b030fad7" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 35287 } health_report { overall_health: HEALTHY } } }
I20260925 15:24:19.893427 22483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:19.890420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43982 (local address 127.3.211.193:35287)
0925 15:24:19.890489 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:19.890491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:19.890501 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:24:19.890550 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:19.890551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:19.890573 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:19.890638 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:24:19.890852 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:19.891371 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:19.892207 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:19.892499 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:19.892564 (+ 65us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:19.892836 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:24:19.892943 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:24:19.893049 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:19.893293 (+ 244us) server_negotiation.cc:299] Negotiation successful
0925 15:24:19.893308 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 15:24:19.895576 22287 raft_consensus.cc:1261] T 8627689917d84f13a5d2f06127bc47e0 P eda9ed42311940eba66ed22f4af810a4 [term 1 FOLLOWER]: Refusing update from remote peer 864e26e1ad044f80be9962f1b030fad7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:24:19.895607 22421 raft_consensus.cc:1261] T 8627689917d84f13a5d2f06127bc47e0 P a46aa5ae2fe844b79c9e5999697b4469 [term 1 FOLLOWER]: Refusing update from remote peer 864e26e1ad044f80be9962f1b030fad7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:24:19.896034 22492 consensus_queue.cc:1035] T 8627689917d84f13a5d2f06127bc47e0 P 864e26e1ad044f80be9962f1b030fad7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a46aa5ae2fe844b79c9e5999697b4469" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 34023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:24:19.896250 22495 consensus_queue.cc:1035] T 8627689917d84f13a5d2f06127bc47e0 P 864e26e1ad044f80be9962f1b030fad7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eda9ed42311940eba66ed22f4af810a4" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 39703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:24:19.902024 22008 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38098:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:24:19.902163 22008 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38098:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:24:19.903079 22008 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7b7abceb716d46f3b7fc4a70f2136e6b: Sending DeleteTablet for 3 replicas of tablet 8627689917d84f13a5d2f06127bc47e0
I20260925 15:24:19.903501 22401 tablet_service.cc:1514] Processing DeleteTablet for tablet 8627689917d84f13a5d2f06127bc47e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:24:19 UTC) from {username='kudu', principal='kudu/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:32836
I20260925 15:24:19.903554 22133 tablet_service.cc:1514] Processing DeleteTablet for tablet 8627689917d84f13a5d2f06127bc47e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:24:19 UTC) from {username='kudu', principal='kudu/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:43972
I20260925 15:24:19.903788 22508 tablet_replica.cc:330] T 8627689917d84f13a5d2f06127bc47e0 P a46aa5ae2fe844b79c9e5999697b4469: stopping tablet replica
I20260925 15:24:19.903786 22267 tablet_service.cc:1514] Processing DeleteTablet for tablet 8627689917d84f13a5d2f06127bc47e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:24:19 UTC) from {username='kudu', principal='kudu/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:54550
I20260925 15:24:19.903911 22508 raft_consensus.cc:2229] T 8627689917d84f13a5d2f06127bc47e0 P a46aa5ae2fe844b79c9e5999697b4469 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:24:19.904038 22508 raft_consensus.cc:2258] T 8627689917d84f13a5d2f06127bc47e0 P a46aa5ae2fe844b79c9e5999697b4469 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:24:19.904043 22510 tablet_replica.cc:330] T 8627689917d84f13a5d2f06127bc47e0 P eda9ed42311940eba66ed22f4af810a4: stopping tablet replica
I20260925 15:24:19.904296 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 22070
I20260925 15:24:19.904244 22509 tablet_replica.cc:330] T 8627689917d84f13a5d2f06127bc47e0 P 864e26e1ad044f80be9962f1b030fad7: stopping tablet replica
I20260925 15:24:19.905342 22508 ts_tablet_manager.cc:1905] T 8627689917d84f13a5d2f06127bc47e0 P a46aa5ae2fe844b79c9e5999697b4469: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:24:19.907024 22508 ts_tablet_manager.cc:1918] T 8627689917d84f13a5d2f06127bc47e0 P a46aa5ae2fe844b79c9e5999697b4469: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:24:19.907327 22508 log.cc:1196] T 8627689917d84f13a5d2f06127bc47e0 P a46aa5ae2fe844b79c9e5999697b4469: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790349610595785-3919-0/minicluster-data/ts-2/wal/wals/8627689917d84f13a5d2f06127bc47e0
I20260925 15:24:19.907687 22508 ts_tablet_manager.cc:1939] T 8627689917d84f13a5d2f06127bc47e0 P a46aa5ae2fe844b79c9e5999697b4469: Deleting consensus metadata
I20260925 15:24:19.908537 21992 catalog_manager.cc:4841] TS a46aa5ae2fe844b79c9e5999697b4469 (127.3.211.195:34023): tablet 8627689917d84f13a5d2f06127bc47e0 (table test-table [id=81a2e154c1cb47e49d42ea04e9719021]) successfully deleted
I20260925 15:24:19.904232 22510 raft_consensus.cc:2229] T 8627689917d84f13a5d2f06127bc47e0 P eda9ed42311940eba66ed22f4af810a4 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260925 15:24:19.911141 21993 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.211.193:35287 (error 108)
I20260925 15:24:19.911315 22510 raft_consensus.cc:2258] T 8627689917d84f13a5d2f06127bc47e0 P eda9ed42311940eba66ed22f4af810a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:24:19.911796 22481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:19.911644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:35287 (local address 127.0.0.1:43992)
0925 15:24:19.911707 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:19.911752 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.193:35287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 15:24:19.911921 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 22204
W20260925 15:24:19.911933 21993 catalog_manager.cc:4584] TS 864e26e1ad044f80be9962f1b030fad7 (127.3.211.193:35287): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8627689917d84f13a5d2f06127bc47e0: Network error: Client connection negotiation failed: client connection to 127.3.211.193:35287: connect: Connection refused (error 111)
I20260925 15:24:19.917585 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 22338
I20260925 15:24:19.917807 22481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:19.917658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.194:39703 (local address 127.0.0.1:54566)
0925 15:24:19.917717 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:19.917755 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.194:39703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 15:24:19.923282 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 21976
2026-09-25T15:24:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2257 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T15:24:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:24:19Z Disabled control of system clock
I20260925 15:24:19.953464 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:43621
--webserver_interface=127.3.211.254
--webserver_port=0
--builtin_ntp_servers=127.3.211.212:43769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.211.254:43621
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:24:20.056897 22515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:20.057245 22515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:20.057276 22515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:20.058707 22515 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:24:20.058773 22515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:20.058787 22515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:20.058800 22515 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:24:20.058813 22515 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:24:20.060493 22515 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:43769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.211.254:43621
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:43621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.211.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:24:20.060920 22515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:20.061391 22515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:20.068377 22520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:20.068377 22523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:20.068377 22521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:20.068661 22515 server_base.cc:1034] running on GCE node
I20260925 15:24:20.069022 22515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:20.069343 22515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:20.070567 22515 hybrid_clock.cc:648] HybridClock initialized: now 1790349860070535 us; error 43 us; skew 500 ppm
I20260925 15:24:20.071998 22515 webserver.cc:460] Webserver started at http://127.3.211.254:41767/ using document root <none> and password file <none>
I20260925 15:24:20.072280 22515 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:20.072378 22515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:20.072542 22515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:20.073726 22515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/master-0/data/instance:
uuid: "3462207ce207412e991251436cce6061"
format_stamp: "Formatted at 2026-09-25 15:24:20 on dist-test-slave-4bk3"
server_key: "def01588da374f37e1160961baef750d"
server_key_iv: "544c279ad45b54a9a93495bebaf2a7a7"
server_key_version: "encryptionkey@0"
I20260925 15:24:20.074122 22515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/master-0/wal/instance:
uuid: "3462207ce207412e991251436cce6061"
format_stamp: "Formatted at 2026-09-25 15:24:20 on dist-test-slave-4bk3"
server_key: "def01588da374f37e1160961baef750d"
server_key_iv: "544c279ad45b54a9a93495bebaf2a7a7"
server_key_version: "encryptionkey@0"
I20260925 15:24:20.075625 22515 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 15:24:20.076617 22529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:20.076974 22515 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:24:20.077117 22515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/master-0/wal
uuid: "3462207ce207412e991251436cce6061"
format_stamp: "Formatted at 2026-09-25 15:24:20 on dist-test-slave-4bk3"
server_key: "def01588da374f37e1160961baef750d"
server_key_iv: "544c279ad45b54a9a93495bebaf2a7a7"
server_key_version: "encryptionkey@0"
I20260925 15:24:20.077206 22515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:20.088011 22515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:20.088470 22515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:20.088636 22515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:20.093600 22515 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.254:43621
I20260925 15:24:20.093683 22581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.254:43621 every 8 connection(s)
I20260925 15:24:20.094072 22515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
I20260925 15:24:20.095176 22582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:20.095827 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 22515
I20260925 15:24:20.095969 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/master-0/wal/instance
I20260925 15:24:20.096194 3919 external_mini_cluster.cc:1428] Setting key f4da3fa2f01d651dcb3c234b90c55f27
I20260925 15:24:20.099004 22582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3462207ce207412e991251436cce6061: Bootstrap starting.
I20260925 15:24:20.099749 22582 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3462207ce207412e991251436cce6061: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:20.100054 22582 log.cc:824] T 00000000000000000000000000000000 P 3462207ce207412e991251436cce6061: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:20.100929 22582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3462207ce207412e991251436cce6061: No bootstrap required, opened a new log
I20260925 15:24:20.102818 22585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:20.096834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42210 (local address 127.3.211.254:43621)
0925 15:24:20.097082 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:20.097091 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:20.097127 (+ 36us) server_negotiation.cc:407] Connection header received
0925 15:24:20.097676 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:20.097685 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:20.097766 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:20.097965 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:20.098215 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:20.099547 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:20.100491 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:20.101103 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:20.101228 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:20.102024 (+ 796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:20.102030 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:20.102030 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:24:20.102234 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:20.102279 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:24:20.102293 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:20.102365 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:20.102511 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 15:24:20.102599 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":88,"threads_started":1}
I20260925 15:24:20.103626 22582 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3462207ce207412e991251436cce6061 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3462207ce207412e991251436cce6061" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 43621 } }
I20260925 15:24:20.104010 22582 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3462207ce207412e991251436cce6061 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:24:20.104280 22582 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3462207ce207412e991251436cce6061 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3462207ce207412e991251436cce6061, State: Initialized, Role: FOLLOWER
I20260925 15:24:20.104517 22582 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3462207ce207412e991251436cce6061 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3462207ce207412e991251436cce6061" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 43621 } }
I20260925 15:24:20.104630 22582 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3462207ce207412e991251436cce6061 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:24:20.104655 22582 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3462207ce207412e991251436cce6061 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:24:20.104715 22582 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3462207ce207412e991251436cce6061 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:20.105348 22582 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3462207ce207412e991251436cce6061 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3462207ce207412e991251436cce6061" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 43621 } }
I20260925 15:24:20.105448 22582 leader_election.cc:302] T 00000000000000000000000000000000 P 3462207ce207412e991251436cce6061 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3462207ce207412e991251436cce6061; no voters:
I20260925 15:24:20.105595 22582 leader_election.cc:288] T 00000000000000000000000000000000 P 3462207ce207412e991251436cce6061 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:24:20.105713 22587 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3462207ce207412e991251436cce6061 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:24:20.105876 22582 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3462207ce207412e991251436cce6061 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:24:20.105896 22587 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3462207ce207412e991251436cce6061 [term 1 LEADER]: Becoming Leader. State: Replica: 3462207ce207412e991251436cce6061, State: Running, Role: LEADER
I20260925 15:24:20.105986 22587 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3462207ce207412e991251436cce6061 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3462207ce207412e991251436cce6061" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 43621 } }
I20260925 15:24:20.106312 22589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3462207ce207412e991251436cce6061 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3462207ce207412e991251436cce6061. Latest consensus state: current_term: 1 leader_uuid: "3462207ce207412e991251436cce6061" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3462207ce207412e991251436cce6061" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 43621 } } }
I20260925 15:24:20.106407 22589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3462207ce207412e991251436cce6061 [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:20.106323 22588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3462207ce207412e991251436cce6061 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3462207ce207412e991251436cce6061" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3462207ce207412e991251436cce6061" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 43621 } } }
I20260925 15:24:20.106585 22588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3462207ce207412e991251436cce6061 [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:20.107096 22594 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:24:20.107620 22594 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:24:20.109167 22594 catalog_manager.cc:1347] Generated new cluster ID: 0590603c48a64ec284a85d23b7f40a64
I20260925 15:24:20.109223 22594 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:24:20.124763 22594 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:24:20.125535 22594 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:24:20.131649 22594 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3462207ce207412e991251436cce6061: Generated new TSK 0
I20260925 15:24:20.131889 22594 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:24:20.137427 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.193:0
--local_ip_for_outbound_sockets=127.3.211.193
--webserver_interface=127.3.211.193
--webserver_port=0
--tserver_master_addrs=127.3.211.254:43621
--builtin_ntp_servers=127.3.211.212:43769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:24:20.233796 22606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:20.234089 22606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:20.234133 22606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:20.235422 22606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:20.235510 22606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:20.235558 22606 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.193
I20260925 15:24:20.237380 22606 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:43769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.211.193
--webserver_port=0
--tserver_master_addrs=127.3.211.254:43621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.193
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:24:20.237731 22606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:20.238081 22606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:20.244521 22612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:20.244681 22614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:20.244524 22611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:20.244959 22606 server_base.cc:1034] running on GCE node
I20260925 15:24:20.245214 22606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:20.245565 22606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:20.246767 22606 hybrid_clock.cc:648] HybridClock initialized: now 1790349860246740 us; error 45 us; skew 500 ppm
I20260925 15:24:20.248414 22606 webserver.cc:460] Webserver started at http://127.3.211.193:44947/ using document root <none> and password file <none>
I20260925 15:24:20.248711 22606 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:20.248773 22606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:20.248901 22606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:20.250342 22606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/ts-0/data/instance:
uuid: "232dc2f709004f359b2bd0bd56b1acec"
format_stamp: "Formatted at 2026-09-25 15:24:20 on dist-test-slave-4bk3"
server_key: "9dd962b6e2a67b5b1e704278793e216b"
server_key_iv: "884b26c77ab7d367a3f86e5d114c6a4a"
server_key_version: "encryptionkey@0"
I20260925 15:24:20.250758 22606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/ts-0/wal/instance:
uuid: "232dc2f709004f359b2bd0bd56b1acec"
format_stamp: "Formatted at 2026-09-25 15:24:20 on dist-test-slave-4bk3"
server_key: "9dd962b6e2a67b5b1e704278793e216b"
server_key_iv: "884b26c77ab7d367a3f86e5d114c6a4a"
server_key_version: "encryptionkey@0"
I20260925 15:24:20.252544 22606 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260925 15:24:20.253647 22620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:20.253916 22606 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:24:20.254030 22606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/ts-0/wal
uuid: "232dc2f709004f359b2bd0bd56b1acec"
format_stamp: "Formatted at 2026-09-25 15:24:20 on dist-test-slave-4bk3"
server_key: "9dd962b6e2a67b5b1e704278793e216b"
server_key_iv: "884b26c77ab7d367a3f86e5d114c6a4a"
server_key_version: "encryptionkey@0"
I20260925 15:24:20.254127 22606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:20.285187 22606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:20.285609 22606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:20.285789 22606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:20.286070 22606 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:24:20.286510 22606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:24:20.286569 22606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:20.286617 22606 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:24:20.286646 22606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:20.294034 22606 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.193:36881
I20260925 15:24:20.294184 22733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.193:36881 every 8 connection(s)
I20260925 15:24:20.294602 22606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:24:20.296793 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 22606
I20260925 15:24:20.296904 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/ts-0/wal/instance
I20260925 15:24:20.297413 3919 external_mini_cluster.cc:1428] Setting key b7f3489cc88c5171345a685253140b41
I20260925 15:24:20.298599 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.194:0
--local_ip_for_outbound_sockets=127.3.211.194
--webserver_interface=127.3.211.194
--webserver_port=0
--tserver_master_addrs=127.3.211.254:43621
--builtin_ntp_servers=127.3.211.212:43769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 15:24:20.301502 22585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:20.295546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:33209 (local address 127.3.211.254:43621)
0925 15:24:20.295617 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:20.295619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:20.295860 (+ 241us) server_negotiation.cc:407] Connection header received
0925 15:24:20.296546 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:20.296548 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:20.296565 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:20.296623 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:20.297429 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:20.298070 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:20.299654 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:20.299868 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:20.300008 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:20.301160 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:20.301162 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:20.301162 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:24:20.301168 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:20.301185 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:24:20.301189 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:20.301267 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:20.301371 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 15:24:20.301417 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 15:24:20.301759 22736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:20.295553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:43621 (local address 127.3.211.193:33209)
0925 15:24:20.295733 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:20.295770 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:20.296419 (+ 649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:20.296713 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:20.296721 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:20.296730 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:24:20.297334 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:20.297340 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:20.298187 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:20.298188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:20.299537 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:20.299541 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:20.300033 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:20.300035 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:20.300189 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:20.300955 (+ 766us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:24:20.300956 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:24:20.300988 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:24:20.301017 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:24:20.301269 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:24:20.301276 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:24:20.301280 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:24:20.301376 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 15:24:20.301513 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":73,"threads_started":1}
I20260925 15:24:20.302322 22734 heartbeater.cc:347] Connected to a master server at 127.3.211.254:43621
I20260925 15:24:20.302448 22734 heartbeater.cc:464] Registering TS with master...
I20260925 15:24:20.302683 22734 heartbeater.cc:511] Master 127.3.211.254:43621 requested a full tablet report, sending...
I20260925 15:24:20.303548 22544 ts_manager.cc:194] Registered new tserver with Master: 232dc2f709004f359b2bd0bd56b1acec (127.3.211.193:36881)
I20260925 15:24:20.304214 22544 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.211.193:33209
W20260925 15:24:20.385342 22737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:20.385649 22737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:20.385691 22737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:20.387555 22737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:20.387727 22737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:20.387789 22737 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.194
I20260925 15:24:20.390231 22737 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:43769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.211.194
--webserver_port=0
--tserver_master_addrs=127.3.211.254:43621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.194
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:24:20.390681 22737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:20.391142 22737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:20.397615 22743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:20.397615 22745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:20.397750 22737 server_base.cc:1034] running on GCE node
W20260925 15:24:20.397614 22742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:20.398123 22737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:20.398419 22737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:20.399626 22737 hybrid_clock.cc:648] HybridClock initialized: now 1790349860399573 us; error 73 us; skew 500 ppm
I20260925 15:24:20.401283 22737 webserver.cc:460] Webserver started at http://127.3.211.194:32845/ using document root <none> and password file <none>
I20260925 15:24:20.401618 22737 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:20.401713 22737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:20.401901 22737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:20.403189 22737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/ts-1/data/instance:
uuid: "709f8d96778c437c93b29d0c1a2be1de"
format_stamp: "Formatted at 2026-09-25 15:24:20 on dist-test-slave-4bk3"
server_key: "325c28d23c02fdc391737ee225d0fa45"
server_key_iv: "7d2630700869e85a0c3dac6ec6c73207"
server_key_version: "encryptionkey@0"
I20260925 15:24:20.403661 22737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/ts-1/wal/instance:
uuid: "709f8d96778c437c93b29d0c1a2be1de"
format_stamp: "Formatted at 2026-09-25 15:24:20 on dist-test-slave-4bk3"
server_key: "325c28d23c02fdc391737ee225d0fa45"
server_key_iv: "7d2630700869e85a0c3dac6ec6c73207"
server_key_version: "encryptionkey@0"
I20260925 15:24:20.405570 22737 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260925 15:24:20.406561 22751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:20.406814 22737 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:24:20.406939 22737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/ts-1/wal
uuid: "709f8d96778c437c93b29d0c1a2be1de"
format_stamp: "Formatted at 2026-09-25 15:24:20 on dist-test-slave-4bk3"
server_key: "325c28d23c02fdc391737ee225d0fa45"
server_key_iv: "7d2630700869e85a0c3dac6ec6c73207"
server_key_version: "encryptionkey@0"
I20260925 15:24:20.407029 22737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:20.417956 22737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:20.418336 22737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:20.418497 22737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:20.418792 22737 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:24:20.419152 22737 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:24:20.419184 22737 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:20.419209 22737 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:24:20.419257 22737 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:20.425108 22737 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.194:41081
I20260925 15:24:20.425256 22864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.194:41081 every 8 connection(s)
I20260925 15:24:20.425545 22737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
I20260925 15:24:20.431612 22585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:20.426368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:44113 (local address 127.3.211.254:43621)
0925 15:24:20.426482 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:20.426486 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:20.426688 (+ 202us) server_negotiation.cc:407] Connection header received
0925 15:24:20.427291 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:20.427292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:20.427305 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:20.427364 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:20.428156 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:20.428825 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:20.429957 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:20.430198 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:20.430290 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:20.431271 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:20.431272 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:20.431272 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:24:20.431279 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:20.431296 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:24:20.431300 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:20.431380 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:20.431485 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 15:24:20.431522 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 15:24:20.431865 22867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:20.426280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:43621 (local address 127.3.211.194:44113)
0925 15:24:20.426556 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:20.426594 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:20.427172 (+ 578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:20.427471 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:20.427474 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:20.427483 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:24:20.428048 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:20.428052 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:20.428950 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:20.428951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:20.429827 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:20.429830 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:20.430299 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:20.430300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:20.430452 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:20.431092 (+ 640us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:24:20.431093 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:24:20.431124 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:24:20.431158 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:24:20.431381 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:24:20.431387 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:24:20.431390 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:24:20.431484 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 15:24:20.431596 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":128,"threads_started":1}
I20260925 15:24:20.432476 22865 heartbeater.cc:347] Connected to a master server at 127.3.211.254:43621
I20260925 15:24:20.432610 22865 heartbeater.cc:464] Registering TS with master...
I20260925 15:24:20.432854 22865 heartbeater.cc:511] Master 127.3.211.254:43621 requested a full tablet report, sending...
I20260925 15:24:20.433364 22544 ts_manager.cc:194] Registered new tserver with Master: 709f8d96778c437c93b29d0c1a2be1de (127.3.211.194:41081)
I20260925 15:24:20.433735 22544 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.211.194:44113
I20260925 15:24:20.434666 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 22737
I20260925 15:24:20.434757 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/ts-1/wal/instance
I20260925 15:24:20.434930 3919 external_mini_cluster.cc:1428] Setting key 187602f81628d7e9bb5954c80ffad06f
I20260925 15:24:20.435948 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.195:0
--local_ip_for_outbound_sockets=127.3.211.195
--webserver_interface=127.3.211.195
--webserver_port=0
--tserver_master_addrs=127.3.211.254:43621
--builtin_ntp_servers=127.3.211.212:43769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:24:20.524266 22868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:20.524730 22868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:20.524758 22868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:20.526070 22868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:20.526125 22868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:20.526156 22868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.195
I20260925 15:24:20.527629 22868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:43769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.211.195
--webserver_port=0
--tserver_master_addrs=127.3.211.254:43621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.195
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:24:20.527874 22868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:20.528164 22868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:20.533918 22873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:20.534155 22868 server_base.cc:1034] running on GCE node
W20260925 15:24:20.533914 22876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:20.533973 22874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:20.534608 22868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:20.534852 22868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:20.536010 22868 hybrid_clock.cc:648] HybridClock initialized: now 1790349860535964 us; error 80 us; skew 500 ppm
I20260925 15:24:20.537602 22868 webserver.cc:460] Webserver started at http://127.3.211.195:43167/ using document root <none> and password file <none>
I20260925 15:24:20.537820 22868 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:20.537869 22868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:20.537972 22868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:20.539031 22868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/ts-2/data/instance:
uuid: "f50840c5ce6249d6a4731a90e2b84fe0"
format_stamp: "Formatted at 2026-09-25 15:24:20 on dist-test-slave-4bk3"
server_key: "88d93d1d745ee7fefc356f169f90af6e"
server_key_iv: "befa55435516a386dd5b435130ff0f8d"
server_key_version: "encryptionkey@0"
I20260925 15:24:20.539351 22868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/ts-2/wal/instance:
uuid: "f50840c5ce6249d6a4731a90e2b84fe0"
format_stamp: "Formatted at 2026-09-25 15:24:20 on dist-test-slave-4bk3"
server_key: "88d93d1d745ee7fefc356f169f90af6e"
server_key_iv: "befa55435516a386dd5b435130ff0f8d"
server_key_version: "encryptionkey@0"
I20260925 15:24:20.541069 22868 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260925 15:24:20.542068 22882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:20.542390 22868 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:24:20.542507 22868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/ts-2/wal
uuid: "f50840c5ce6249d6a4731a90e2b84fe0"
format_stamp: "Formatted at 2026-09-25 15:24:20 on dist-test-slave-4bk3"
server_key: "88d93d1d745ee7fefc356f169f90af6e"
server_key_iv: "befa55435516a386dd5b435130ff0f8d"
server_key_version: "encryptionkey@0"
I20260925 15:24:20.542618 22868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:20.561398 22868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:20.561805 22868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:20.561975 22868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:20.562273 22868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:24:20.562647 22868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:24:20.562685 22868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:20.562734 22868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:24:20.562767 22868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:20.569484 22868 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.195:36487
I20260925 15:24:20.569589 22995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.195:36487 every 8 connection(s)
I20260925 15:24:20.570045 22868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
I20260925 15:24:20.572395 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 22868
I20260925 15:24:20.572525 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/ts-2/wal/instance
I20260925 15:24:20.572760 3919 external_mini_cluster.cc:1428] Setting key a2f317375e74cdd4d61f453cb5ba8544
I20260925 15:24:20.576663 22585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:20.571164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:60843 (local address 127.3.211.254:43621)
0925 15:24:20.571249 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:20.571251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:20.571460 (+ 209us) server_negotiation.cc:407] Connection header received
0925 15:24:20.572052 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:20.572054 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:20.572066 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:20.572125 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:20.573235 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:20.573955 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:20.575069 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:20.575291 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:20.575380 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:20.576354 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:20.576356 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:20.576356 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:24:20.576361 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:20.576379 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:24:20.576384 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:20.576439 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:20.576572 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 15:24:20.576597 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 15:24:20.576951 22998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:20.570910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:43621 (local address 127.3.211.195:60843)
0925 15:24:20.571244 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:20.571308 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:20.571932 (+ 624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:20.572221 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:20.572226 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:20.572237 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:24:20.573130 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:20.573136 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:20.574055 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:20.574056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:20.574965 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:20.574968 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:20.575400 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:20.575401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:20.575542 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:20.576146 (+ 604us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:24:20.576146 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:24:20.576176 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:24:20.576201 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:24:20.576463 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:24:20.576477 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:24:20.576481 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:24:20.576561 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 15:24:20.576657 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":80,"threads_started":1}
I20260925 15:24:20.577694 22996 heartbeater.cc:347] Connected to a master server at 127.3.211.254:43621
I20260925 15:24:20.577795 22996 heartbeater.cc:464] Registering TS with master...
I20260925 15:24:20.577975 22996 heartbeater.cc:511] Master 127.3.211.254:43621 requested a full tablet report, sending...
I20260925 15:24:20.578429 22544 ts_manager.cc:194] Registered new tserver with Master: f50840c5ce6249d6a4731a90e2b84fe0 (127.3.211.195:36487)
I20260925 15:24:20.578779 22544 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.211.195:60843
I20260925 15:24:20.585541 3919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:24:20.591081 22585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:20.587488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42214 (local address 127.3.211.254:43621)
0925 15:24:20.587583 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:20.587586 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:20.587704 (+ 118us) server_negotiation.cc:407] Connection header received
0925 15:24:20.587794 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:20.587795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:20.587806 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:20.587893 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:20.588096 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:20.588713 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:20.589579 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:20.589797 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:20.589882 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:20.590762 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:20.590764 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:20.590764 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:24:20.590771 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:20.590790 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:24:20.590794 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:20.590897 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:20.590967 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 15:24:20.590979 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 15:24:20.593632 22544 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42214:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:24:20.595214 22544 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:24:20.602994 23006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:20.598022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.194:41081 (local address 127.0.0.1:58344)
0925 15:24:20.598650 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:20.598661 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:20.598709 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:20.598998 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:20.598999 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:20.599001 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:24:20.599052 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:20.599053 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:20.599984 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:20.599985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:20.601364 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:20.601367 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:20.602622 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:20.602623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:20.602722 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:20.602736 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 15:24:20.602818 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 15:24:20.602881 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"mutex_wait_us":6,"spinlock_wait_cycles":896,"thread_start_us":95,"threads_started":1}
I20260925 15:24:20.603011 23009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:20.598407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58344 (local address 127.3.211.194:41081)
0925 15:24:20.598608 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:20.598613 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:20.598771 (+ 158us) server_negotiation.cc:407] Connection header received
0925 15:24:20.598830 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:20.598832 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:20.598849 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:20.598929 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:24:20.599119 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:20.599864 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:20.601492 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:20.602417 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:20.602480 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:20.602581 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:20.602862 (+ 281us) server_negotiation.cc:299] Negotiation successful
0925 15:24:20.602911 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":94,"threads_started":1}
I20260925 15:24:20.603505 23004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:20.598021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:36881 (local address 127.0.0.1:34452)
0925 15:24:20.598231 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:20.598276 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:20.598353 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:20.598677 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:20.598680 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:20.598683 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:24:20.598758 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:20.598761 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:20.600451 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:20.600452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:20.601503 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:20.601505 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:20.602623 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:20.602624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:20.602893 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:20.602898 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:24:20.602947 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 15:24:20.602981 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":95,"threads_started":1}
I20260925 15:24:20.603607 23005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:20.598022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.195:36487 (local address 127.0.0.1:50880)
0925 15:24:20.598483 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:20.598493 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:20.598536 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:20.598770 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:20.598771 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:20.598773 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:24:20.598807 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:20.598808 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:20.601628 (+ 2820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:20.601629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:20.602324 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:20.602326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:20.603341 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:20.603343 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:20.603417 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:20.603421 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:24:20.603480 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 15:24:20.603530 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":87,"threads_started":1}
I20260925 15:24:20.603724 23007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:20.598199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34452 (local address 127.3.211.193:36881)
0925 15:24:20.598405 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:20.598410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:20.598423 (+ 13us) server_negotiation.cc:407] Connection header received
0925 15:24:20.598467 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:20.598470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:20.598485 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:20.598572 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:24:20.598856 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:20.600144 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:20.601609 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:20.602395 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:20.602480 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:20.602582 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:20.603040 (+ 458us) server_negotiation.cc:299] Negotiation successful
0925 15:24:20.603094 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":116,"threads_started":1}
I20260925 15:24:20.603904 23008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:20.598326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50880 (local address 127.3.211.195:36487)
0925 15:24:20.598513 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:20.598517 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:20.598527 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:24:20.598623 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:20.598624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:20.598639 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:20.598712 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:24:20.598882 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:20.600167 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:20.602444 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:20.603093 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:20.603181 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:20.603272 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:20.603730 (+ 458us) server_negotiation.cc:299] Negotiation successful
0925 15:24:20.603800 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":90,"threads_started":1}
I20260925 15:24:20.604127 22799 tablet_service.cc:1467] Processing CreateTablet for tablet 75b9d3c738244b56b237b6becba7856e (DEFAULT_TABLE table=test-table [id=1d1dfebb15b74df796b172ef3be83faa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:24:20.604411 22668 tablet_service.cc:1467] Processing CreateTablet for tablet 75b9d3c738244b56b237b6becba7856e (DEFAULT_TABLE table=test-table [id=1d1dfebb15b74df796b172ef3be83faa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:24:20.604604 22799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75b9d3c738244b56b237b6becba7856e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:20.604612 22930 tablet_service.cc:1467] Processing CreateTablet for tablet 75b9d3c738244b56b237b6becba7856e (DEFAULT_TABLE table=test-table [id=1d1dfebb15b74df796b172ef3be83faa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:24:20.604765 22668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75b9d3c738244b56b237b6becba7856e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:20.604935 22930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75b9d3c738244b56b237b6becba7856e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:20.608853 23010 tablet_bootstrap.cc:492] T 75b9d3c738244b56b237b6becba7856e P 232dc2f709004f359b2bd0bd56b1acec: Bootstrap starting.
I20260925 15:24:20.609001 23011 tablet_bootstrap.cc:492] T 75b9d3c738244b56b237b6becba7856e P f50840c5ce6249d6a4731a90e2b84fe0: Bootstrap starting.
I20260925 15:24:20.609313 23012 tablet_bootstrap.cc:492] T 75b9d3c738244b56b237b6becba7856e P 709f8d96778c437c93b29d0c1a2be1de: Bootstrap starting.
I20260925 15:24:20.609895 23011 tablet_bootstrap.cc:654] T 75b9d3c738244b56b237b6becba7856e P f50840c5ce6249d6a4731a90e2b84fe0: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:20.610152 23010 tablet_bootstrap.cc:654] T 75b9d3c738244b56b237b6becba7856e P 232dc2f709004f359b2bd0bd56b1acec: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:20.610257 23011 log.cc:824] T 75b9d3c738244b56b237b6becba7856e P f50840c5ce6249d6a4731a90e2b84fe0: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:20.610329 23012 tablet_bootstrap.cc:654] T 75b9d3c738244b56b237b6becba7856e P 709f8d96778c437c93b29d0c1a2be1de: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:20.610539 23010 log.cc:824] T 75b9d3c738244b56b237b6becba7856e P 232dc2f709004f359b2bd0bd56b1acec: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:20.610718 23012 log.cc:824] T 75b9d3c738244b56b237b6becba7856e P 709f8d96778c437c93b29d0c1a2be1de: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:20.611310 23011 tablet_bootstrap.cc:492] T 75b9d3c738244b56b237b6becba7856e P f50840c5ce6249d6a4731a90e2b84fe0: No bootstrap required, opened a new log
I20260925 15:24:20.611374 23011 ts_tablet_manager.cc:1397] T 75b9d3c738244b56b237b6becba7856e P f50840c5ce6249d6a4731a90e2b84fe0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:24:20.611515 23010 tablet_bootstrap.cc:492] T 75b9d3c738244b56b237b6becba7856e P 232dc2f709004f359b2bd0bd56b1acec: No bootstrap required, opened a new log
I20260925 15:24:20.611609 23010 ts_tablet_manager.cc:1397] T 75b9d3c738244b56b237b6becba7856e P 232dc2f709004f359b2bd0bd56b1acec: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 15:24:20.612823 23012 tablet_bootstrap.cc:492] T 75b9d3c738244b56b237b6becba7856e P 709f8d96778c437c93b29d0c1a2be1de: No bootstrap required, opened a new log
I20260925 15:24:20.612911 23012 ts_tablet_manager.cc:1397] T 75b9d3c738244b56b237b6becba7856e P 709f8d96778c437c93b29d0c1a2be1de: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 15:24:20.614092 23011 raft_consensus.cc:348] T 75b9d3c738244b56b237b6becba7856e P f50840c5ce6249d6a4731a90e2b84fe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709f8d96778c437c93b29d0c1a2be1de" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 41081 } } peers { permanent_uuid: "232dc2f709004f359b2bd0bd56b1acec" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 36881 } } peers { permanent_uuid: "f50840c5ce6249d6a4731a90e2b84fe0" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 36487 } }
I20260925 15:24:20.614280 23011 raft_consensus.cc:374] T 75b9d3c738244b56b237b6becba7856e P f50840c5ce6249d6a4731a90e2b84fe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:24:20.614326 23011 raft_consensus.cc:729] T 75b9d3c738244b56b237b6becba7856e P f50840c5ce6249d6a4731a90e2b84fe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f50840c5ce6249d6a4731a90e2b84fe0, State: Initialized, Role: FOLLOWER
I20260925 15:24:20.614394 23010 raft_consensus.cc:348] T 75b9d3c738244b56b237b6becba7856e P 232dc2f709004f359b2bd0bd56b1acec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709f8d96778c437c93b29d0c1a2be1de" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 41081 } } peers { permanent_uuid: "232dc2f709004f359b2bd0bd56b1acec" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 36881 } } peers { permanent_uuid: "f50840c5ce6249d6a4731a90e2b84fe0" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 36487 } }
I20260925 15:24:20.614579 23010 raft_consensus.cc:374] T 75b9d3c738244b56b237b6becba7856e P 232dc2f709004f359b2bd0bd56b1acec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:24:20.614619 23010 raft_consensus.cc:729] T 75b9d3c738244b56b237b6becba7856e P 232dc2f709004f359b2bd0bd56b1acec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 232dc2f709004f359b2bd0bd56b1acec, State: Initialized, Role: FOLLOWER
I20260925 15:24:20.614735 23010 consensus_queue.cc:260] T 75b9d3c738244b56b237b6becba7856e P 232dc2f709004f359b2bd0bd56b1acec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709f8d96778c437c93b29d0c1a2be1de" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 41081 } } peers { permanent_uuid: "232dc2f709004f359b2bd0bd56b1acec" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 36881 } } peers { permanent_uuid: "f50840c5ce6249d6a4731a90e2b84fe0" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 36487 } }
I20260925 15:24:20.614668 23011 consensus_queue.cc:260] T 75b9d3c738244b56b237b6becba7856e P f50840c5ce6249d6a4731a90e2b84fe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709f8d96778c437c93b29d0c1a2be1de" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 41081 } } peers { permanent_uuid: "232dc2f709004f359b2bd0bd56b1acec" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 36881 } } peers { permanent_uuid: "f50840c5ce6249d6a4731a90e2b84fe0" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 36487 } }
I20260925 15:24:20.615057 23010 ts_tablet_manager.cc:1428] T 75b9d3c738244b56b237b6becba7856e P 232dc2f709004f359b2bd0bd56b1acec: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260925 15:24:20.615190 22734 heartbeater.cc:503] Master 127.3.211.254:43621 was elected leader, sending a full tablet report...
I20260925 15:24:20.615177 23011 ts_tablet_manager.cc:1428] T 75b9d3c738244b56b237b6becba7856e P f50840c5ce6249d6a4731a90e2b84fe0: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260925 15:24:20.615499 22996 heartbeater.cc:503] Master 127.3.211.254:43621 was elected leader, sending a full tablet report...
I20260925 15:24:20.615718 23012 raft_consensus.cc:348] T 75b9d3c738244b56b237b6becba7856e P 709f8d96778c437c93b29d0c1a2be1de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709f8d96778c437c93b29d0c1a2be1de" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 41081 } } peers { permanent_uuid: "232dc2f709004f359b2bd0bd56b1acec" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 36881 } } peers { permanent_uuid: "f50840c5ce6249d6a4731a90e2b84fe0" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 36487 } }
I20260925 15:24:20.615916 23012 raft_consensus.cc:374] T 75b9d3c738244b56b237b6becba7856e P 709f8d96778c437c93b29d0c1a2be1de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:24:20.615957 23012 raft_consensus.cc:729] T 75b9d3c738244b56b237b6becba7856e P 709f8d96778c437c93b29d0c1a2be1de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 709f8d96778c437c93b29d0c1a2be1de, State: Initialized, Role: FOLLOWER
I20260925 15:24:20.616098 23012 consensus_queue.cc:260] T 75b9d3c738244b56b237b6becba7856e P 709f8d96778c437c93b29d0c1a2be1de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709f8d96778c437c93b29d0c1a2be1de" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 41081 } } peers { permanent_uuid: "232dc2f709004f359b2bd0bd56b1acec" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 36881 } } peers { permanent_uuid: "f50840c5ce6249d6a4731a90e2b84fe0" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 36487 } }
I20260925 15:24:20.616438 23012 ts_tablet_manager.cc:1428] T 75b9d3c738244b56b237b6becba7856e P 709f8d96778c437c93b29d0c1a2be1de: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260925 15:24:20.616472 22865 heartbeater.cc:503] Master 127.3.211.254:43621 was elected leader, sending a full tablet report...
I20260925 15:24:20.674729 23016 raft_consensus.cc:482] T 75b9d3c738244b56b237b6becba7856e P 232dc2f709004f359b2bd0bd56b1acec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:24:20.674937 23016 raft_consensus.cc:504] T 75b9d3c738244b56b237b6becba7856e P 232dc2f709004f359b2bd0bd56b1acec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709f8d96778c437c93b29d0c1a2be1de" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 41081 } } peers { permanent_uuid: "232dc2f709004f359b2bd0bd56b1acec" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 36881 } } peers { permanent_uuid: "f50840c5ce6249d6a4731a90e2b84fe0" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 36487 } }
I20260925 15:24:20.675393 23016 leader_election.cc:288] T 75b9d3c738244b56b237b6becba7856e P 232dc2f709004f359b2bd0bd56b1acec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 709f8d96778c437c93b29d0c1a2be1de (127.3.211.194:41081), f50840c5ce6249d6a4731a90e2b84fe0 (127.3.211.195:36487)
W20260925 15:24:20.676138 22866 tablet.cc:2369] T 75b9d3c738244b56b237b6becba7856e P 709f8d96778c437c93b29d0c1a2be1de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:24:20.678540 22736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:20.675621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.194:41081 (local address 127.3.211.193:55143)
0925 15:24:20.675738 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:20.675751 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:20.675814 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:20.676024 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:20.676025 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:20.676027 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:24:20.676102 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:20.676104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:20.676850 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:20.676851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:20.677652 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:20.677654 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:20.678229 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:20.678230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:20.678314 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:20.678320 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 15:24:20.678385 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 15:24:20.678442 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":5}
I20260925 15:24:20.678563 23009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:20.675709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:55143 (local address 127.3.211.194:41081)
0925 15:24:20.675753 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:20.675755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:20.675814 (+ 59us) server_negotiation.cc:407] Connection header received
0925 15:24:20.675907 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:20.675908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:20.675924 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:20.675982 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:24:20.676186 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:20.676751 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:20.677740 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:20.678123 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:20.678180 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:20.678186 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:20.678436 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 15:24:20.678487 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 15:24:20.678776 23008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:20.675738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:51907 (local address 127.3.211.195:36487)
0925 15:24:20.675789 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:20.675791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:20.675903 (+ 112us) server_negotiation.cc:407] Connection header received
0925 15:24:20.675954 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:20.675955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:20.675965 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:20.676033 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:24:20.676225 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:20.676761 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:20.677813 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:20.678288 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:20.678385 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:20.678403 (+ 18us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:20.678631 (+ 228us) server_negotiation.cc:299] Negotiation successful
0925 15:24:20.678689 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 15:24:20.678942 23019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:20.675608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.195:36487 (local address 127.3.211.193:51907)
0925 15:24:20.675826 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:20.675836 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:20.675898 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:20.676052 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:20.676053 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:20.676056 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:24:20.676157 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:20.676159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:20.676880 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:20.676881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:20.677683 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:20.677685 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:20.678448 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:20.678449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:20.678532 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:20.678536 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:24:20.678610 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 15:24:20.678785 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":66,"thread_start_us":107,"threads_started":1}
I20260925 15:24:20.679000 22819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75b9d3c738244b56b237b6becba7856e" candidate_uuid: "232dc2f709004f359b2bd0bd56b1acec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "709f8d96778c437c93b29d0c1a2be1de" is_pre_election: true
I20260925 15:24:20.679180 22819 raft_consensus.cc:2454] T 75b9d3c738244b56b237b6becba7856e P 709f8d96778c437c93b29d0c1a2be1de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 232dc2f709004f359b2bd0bd56b1acec in term 0.
I20260925 15:24:20.679306 22950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75b9d3c738244b56b237b6becba7856e" candidate_uuid: "232dc2f709004f359b2bd0bd56b1acec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f50840c5ce6249d6a4731a90e2b84fe0" is_pre_election: true
I20260925 15:24:20.679458 22950 raft_consensus.cc:2454] T 75b9d3c738244b56b237b6becba7856e P f50840c5ce6249d6a4731a90e2b84fe0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 232dc2f709004f359b2bd0bd56b1acec in term 0.
I20260925 15:24:20.679432 22623 leader_election.cc:302] T 75b9d3c738244b56b237b6becba7856e P 232dc2f709004f359b2bd0bd56b1acec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 232dc2f709004f359b2bd0bd56b1acec, 709f8d96778c437c93b29d0c1a2be1de; no voters:
I20260925 15:24:20.679570 23016 raft_consensus.cc:2781] T 75b9d3c738244b56b237b6becba7856e P 232dc2f709004f359b2bd0bd56b1acec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:24:20.679649 23016 raft_consensus.cc:482] T 75b9d3c738244b56b237b6becba7856e P 232dc2f709004f359b2bd0bd56b1acec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:24:20.679677 23016 raft_consensus.cc:3037] T 75b9d3c738244b56b237b6becba7856e P 232dc2f709004f359b2bd0bd56b1acec [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:20.680477 23016 raft_consensus.cc:504] T 75b9d3c738244b56b237b6becba7856e P 232dc2f709004f359b2bd0bd56b1acec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709f8d96778c437c93b29d0c1a2be1de" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 41081 } } peers { permanent_uuid: "232dc2f709004f359b2bd0bd56b1acec" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 36881 } } peers { permanent_uuid: "f50840c5ce6249d6a4731a90e2b84fe0" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 36487 } }
I20260925 15:24:20.680655 23016 leader_election.cc:288] T 75b9d3c738244b56b237b6becba7856e P 232dc2f709004f359b2bd0bd56b1acec [CANDIDATE]: Term 1 election: Requested vote from peers 709f8d96778c437c93b29d0c1a2be1de (127.3.211.194:41081), f50840c5ce6249d6a4731a90e2b84fe0 (127.3.211.195:36487)
I20260925 15:24:20.680847 22819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75b9d3c738244b56b237b6becba7856e" candidate_uuid: "232dc2f709004f359b2bd0bd56b1acec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "709f8d96778c437c93b29d0c1a2be1de"
I20260925 15:24:20.680847 22950 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75b9d3c738244b56b237b6becba7856e" candidate_uuid: "232dc2f709004f359b2bd0bd56b1acec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f50840c5ce6249d6a4731a90e2b84fe0"
I20260925 15:24:20.680939 22819 raft_consensus.cc:3037] T 75b9d3c738244b56b237b6becba7856e P 709f8d96778c437c93b29d0c1a2be1de [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:20.680954 22950 raft_consensus.cc:3037] T 75b9d3c738244b56b237b6becba7856e P f50840c5ce6249d6a4731a90e2b84fe0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:20.681878 22950 raft_consensus.cc:2454] T 75b9d3c738244b56b237b6becba7856e P f50840c5ce6249d6a4731a90e2b84fe0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 232dc2f709004f359b2bd0bd56b1acec in term 1.
I20260925 15:24:20.681878 22819 raft_consensus.cc:2454] T 75b9d3c738244b56b237b6becba7856e P 709f8d96778c437c93b29d0c1a2be1de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 232dc2f709004f359b2bd0bd56b1acec in term 1.
I20260925 15:24:20.682065 22623 leader_election.cc:302] T 75b9d3c738244b56b237b6becba7856e P 232dc2f709004f359b2bd0bd56b1acec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 232dc2f709004f359b2bd0bd56b1acec, 709f8d96778c437c93b29d0c1a2be1de; no voters:
I20260925 15:24:20.682175 23016 raft_consensus.cc:2781] T 75b9d3c738244b56b237b6becba7856e P 232dc2f709004f359b2bd0bd56b1acec [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:24:20.682418 23016 raft_consensus.cc:686] T 75b9d3c738244b56b237b6becba7856e P 232dc2f709004f359b2bd0bd56b1acec [term 1 LEADER]: Becoming Leader. State: Replica: 232dc2f709004f359b2bd0bd56b1acec, State: Running, Role: LEADER
I20260925 15:24:20.682583 23016 consensus_queue.cc:237] T 75b9d3c738244b56b237b6becba7856e P 232dc2f709004f359b2bd0bd56b1acec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709f8d96778c437c93b29d0c1a2be1de" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 41081 } } peers { permanent_uuid: "232dc2f709004f359b2bd0bd56b1acec" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 36881 } } peers { permanent_uuid: "f50840c5ce6249d6a4731a90e2b84fe0" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 36487 } }
I20260925 15:24:20.683568 22546 catalog_manager.cc:5495] T 75b9d3c738244b56b237b6becba7856e P 232dc2f709004f359b2bd0bd56b1acec reported cstate change: term changed from 0 to 1, leader changed from <none> to 232dc2f709004f359b2bd0bd56b1acec (127.3.211.193). New cstate: current_term: 1 leader_uuid: "232dc2f709004f359b2bd0bd56b1acec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "709f8d96778c437c93b29d0c1a2be1de" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 41081 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "232dc2f709004f359b2bd0bd56b1acec" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 36881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f50840c5ce6249d6a4731a90e2b84fe0" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 36487 } health_report { overall_health: UNKNOWN } } }
I20260925 15:24:20.703100 23007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:20.700490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34462 (local address 127.3.211.193:36881)
0925 15:24:20.700560 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:20.700562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:20.700571 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:24:20.700622 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:20.700624 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:20.700646 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:20.700743 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:24:20.700913 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:20.701403 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:20.702220 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:20.702430 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:20.702521 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:20.702684 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:24:20.702785 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:24:20.702906 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:20.702966 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 15:24:20.702979 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 15:24:20.705323 22950 raft_consensus.cc:1261] T 75b9d3c738244b56b237b6becba7856e P f50840c5ce6249d6a4731a90e2b84fe0 [term 1 FOLLOWER]: Refusing update from remote peer 232dc2f709004f359b2bd0bd56b1acec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:24:20.705324 22819 raft_consensus.cc:1261] T 75b9d3c738244b56b237b6becba7856e P 709f8d96778c437c93b29d0c1a2be1de [term 1 FOLLOWER]: Refusing update from remote peer 232dc2f709004f359b2bd0bd56b1acec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:24:20.705912 23020 consensus_queue.cc:1035] T 75b9d3c738244b56b237b6becba7856e P 232dc2f709004f359b2bd0bd56b1acec [LEADER]: Connected to new peer: Peer: permanent_uuid: "709f8d96778c437c93b29d0c1a2be1de" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 41081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:24:20.706194 23016 consensus_queue.cc:1035] T 75b9d3c738244b56b237b6becba7856e P 232dc2f709004f359b2bd0bd56b1acec [LEADER]: Connected to new peer: Peer: permanent_uuid: "f50840c5ce6249d6a4731a90e2b84fe0" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 36487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:24:20.709955 23023 mvcc.cc:205] Tried to move back new op lower bound from 7333273029446062080 to 7333273029356589056. Current Snapshot: MvccSnapshot[applied={T|T < 7333273029446062080}]
I20260925 15:24:20.710495 23026 mvcc.cc:205] Tried to move back new op lower bound from 7333273029446062080 to 7333273029356589056. Current Snapshot: MvccSnapshot[applied={T|T < 7333273029446062080}]
I20260925 15:24:20.712725 22546 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:24:20.712849 22546 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:24:20.713649 22546 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3462207ce207412e991251436cce6061: Sending DeleteTablet for 3 replicas of tablet 75b9d3c738244b56b237b6becba7856e
I20260925 15:24:20.713958 22799 tablet_service.cc:1514] Processing DeleteTablet for tablet 75b9d3c738244b56b237b6becba7856e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:24:20 UTC) from {username='slave'} at 127.0.0.1:58344
I20260925 15:24:20.714007 22668 tablet_service.cc:1514] Processing DeleteTablet for tablet 75b9d3c738244b56b237b6becba7856e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:24:20 UTC) from {username='slave'} at 127.0.0.1:34452
I20260925 15:24:20.714198 23032 tablet_replica.cc:330] T 75b9d3c738244b56b237b6becba7856e P 709f8d96778c437c93b29d0c1a2be1de: stopping tablet replica
I20260925 15:24:20.714299 23032 raft_consensus.cc:2229] T 75b9d3c738244b56b237b6becba7856e P 709f8d96778c437c93b29d0c1a2be1de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:24:20.714395 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 22606
I20260925 15:24:20.714432 22930 tablet_service.cc:1514] Processing DeleteTablet for tablet 75b9d3c738244b56b237b6becba7856e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:24:20 UTC) from {username='slave'} at 127.0.0.1:50880
I20260925 15:24:20.715194 23032 raft_consensus.cc:2258] T 75b9d3c738244b56b237b6becba7856e P 709f8d96778c437c93b29d0c1a2be1de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:24:20.715581 23034 tablet_replica.cc:330] T 75b9d3c738244b56b237b6becba7856e P f50840c5ce6249d6a4731a90e2b84fe0: stopping tablet replica
I20260925 15:24:20.715688 23034 raft_consensus.cc:2229] T 75b9d3c738244b56b237b6becba7856e P f50840c5ce6249d6a4731a90e2b84fe0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:24:20.715737 23032 ts_tablet_manager.cc:1905] T 75b9d3c738244b56b237b6becba7856e P 709f8d96778c437c93b29d0c1a2be1de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:24:20.715782 23034 raft_consensus.cc:2258] T 75b9d3c738244b56b237b6becba7856e P f50840c5ce6249d6a4731a90e2b84fe0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:24:20.716226 23034 ts_tablet_manager.cc:1905] T 75b9d3c738244b56b237b6becba7856e P f50840c5ce6249d6a4731a90e2b84fe0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:24:20.718099 23032 ts_tablet_manager.cc:1918] T 75b9d3c738244b56b237b6becba7856e P 709f8d96778c437c93b29d0c1a2be1de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:24:20.718200 23032 log.cc:1196] T 75b9d3c738244b56b237b6becba7856e P 709f8d96778c437c93b29d0c1a2be1de: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/ts-1/wal/wals/75b9d3c738244b56b237b6becba7856e
I20260925 15:24:20.718276 23034 ts_tablet_manager.cc:1918] T 75b9d3c738244b56b237b6becba7856e P f50840c5ce6249d6a4731a90e2b84fe0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:24:20.718353 23034 log.cc:1196] T 75b9d3c738244b56b237b6becba7856e P f50840c5ce6249d6a4731a90e2b84fe0: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790349610595785-3919-0/minicluster-data/ts-2/wal/wals/75b9d3c738244b56b237b6becba7856e
I20260925 15:24:20.718531 23032 ts_tablet_manager.cc:1939] T 75b9d3c738244b56b237b6becba7856e P 709f8d96778c437c93b29d0c1a2be1de: Deleting consensus metadata
I20260925 15:24:20.718601 23034 ts_tablet_manager.cc:1939] T 75b9d3c738244b56b237b6becba7856e P f50840c5ce6249d6a4731a90e2b84fe0: Deleting consensus metadata
I20260925 15:24:20.718999 22532 catalog_manager.cc:4841] TS 709f8d96778c437c93b29d0c1a2be1de (127.3.211.194:41081): tablet 75b9d3c738244b56b237b6becba7856e (table test-table [id=1d1dfebb15b74df796b172ef3be83faa]) successfully deleted
I20260925 15:24:20.719033 22530 catalog_manager.cc:4841] TS f50840c5ce6249d6a4731a90e2b84fe0 (127.3.211.195:36487): tablet 75b9d3c738244b56b237b6becba7856e (table test-table [id=1d1dfebb15b74df796b172ef3be83faa]) successfully deleted
W20260925 15:24:20.721623 22531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.211.193:36881 (error 108)
I20260925 15:24:20.721812 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 22737
I20260925 15:24:20.722697 23005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:20.722510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:36881 (local address 127.0.0.1:34468)
0925 15:24:20.722581 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:20.722645 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.193:36881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 15:24:20.722851 22531 catalog_manager.cc:4584] TS 232dc2f709004f359b2bd0bd56b1acec (127.3.211.193:36881): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 75b9d3c738244b56b237b6becba7856e: Network error: Client connection negotiation failed: client connection to 127.3.211.193:36881: connect: Connection refused (error 111)
I20260925 15:24:20.728502 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 22868
I20260925 15:24:20.734392 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 22515
2026-09-25T15:24:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (809 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:24:20 dist-test-slave-4bk3 krb5kdc[23038](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:24:20 dist-test-slave-4bk3 krb5kdc[23038](info): set up 2 sockets
Sep 25 15:24:20 dist-test-slave-4bk3 krb5kdc[23038](info): commencing operation
krb5kdc: starting...
W20260925 15:24:21.796434 3919 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Sep 25 15:24:21 dist-test-slave-4bk3 krb5kdc[23038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:24:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:24:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.211.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
I20260925 15:24:21.932641 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:41759
--webserver_interface=127.3.211.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:37007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.211.254:41759
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:24:22.028441 23051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:22.028779 23051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:22.028810 23051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:22.030900 23051 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:24:22.031008 23051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:22.031044 23051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:22.031059 23051 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:24:22.031072 23051 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:24:22.033424 23051 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:37007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.211.254:41759
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:41759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.211.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:24:22.033768 23051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:22.034106 23051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:22.040796 23057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:22.040943 23056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:22.040796 23059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:22.041105 23051 server_base.cc:1034] running on GCE node
I20260925 15:24:22.041711 23051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:22.042045 23051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:22.043524 23051 hybrid_clock.cc:648] HybridClock initialized: now 1790349862043469 us; error 59 us; skew 500 ppm
Sep 25 15:24:22 dist-test-slave-4bk3 krb5kdc[23038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349862, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:24:22.046335 23051 init.cc:377] Logged in from keytab as kudu/127.3.211.254@KRBTEST.COM (short username kudu)
I20260925 15:24:22.046849 23051 webserver.cc:460] Webserver started at http://127.3.211.254:41393/ using document root <none> and password file <none>
I20260925 15:24:22.047106 23051 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:22.047155 23051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:22.047310 23051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:22.048533 23051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/master-0/data/instance:
uuid: "748617836c4b4b669e8dab2e911cf9a2"
format_stamp: "Formatted at 2026-09-25 15:24:22 on dist-test-slave-4bk3"
server_key: "65d2d75dd4d46cdb84b902bd77bbd3e4"
server_key_iv: "4bed60fe0bb02910131304d9a5ecffb6"
server_key_version: "encryptionkey@0"
I20260925 15:24:22.048923 23051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/master-0/wal/instance:
uuid: "748617836c4b4b669e8dab2e911cf9a2"
format_stamp: "Formatted at 2026-09-25 15:24:22 on dist-test-slave-4bk3"
server_key: "65d2d75dd4d46cdb84b902bd77bbd3e4"
server_key_iv: "4bed60fe0bb02910131304d9a5ecffb6"
server_key_version: "encryptionkey@0"
I20260925 15:24:22.050661 23051 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260925 15:24:22.051755 23066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:22.052206 23051 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:24:22.052495 23051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/master-0/wal
uuid: "748617836c4b4b669e8dab2e911cf9a2"
format_stamp: "Formatted at 2026-09-25 15:24:22 on dist-test-slave-4bk3"
server_key: "65d2d75dd4d46cdb84b902bd77bbd3e4"
server_key_iv: "4bed60fe0bb02910131304d9a5ecffb6"
server_key_version: "encryptionkey@0"
I20260925 15:24:22.052587 23051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:22.073738 23051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:22.076109 23051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:22.076265 23051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:22.080586 23051 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.254:41759
I20260925 15:24:22.080621 23118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.254:41759 every 8 connection(s)
I20260925 15:24:22.081001 23051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
I20260925 15:24:22.081965 23119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:22.084312 23119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 748617836c4b4b669e8dab2e911cf9a2: Bootstrap starting.
I20260925 15:24:22.085000 23119 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 748617836c4b4b669e8dab2e911cf9a2: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:22.085294 23119 log.cc:824] T 00000000000000000000000000000000 P 748617836c4b4b669e8dab2e911cf9a2: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:22.086354 23119 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 748617836c4b4b669e8dab2e911cf9a2: No bootstrap required, opened a new log
I20260925 15:24:22.088614 23119 raft_consensus.cc:348] T 00000000000000000000000000000000 P 748617836c4b4b669e8dab2e911cf9a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "748617836c4b4b669e8dab2e911cf9a2" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 41759 } }
I20260925 15:24:22.088877 23119 raft_consensus.cc:374] T 00000000000000000000000000000000 P 748617836c4b4b669e8dab2e911cf9a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:24:22.088919 23119 raft_consensus.cc:729] T 00000000000000000000000000000000 P 748617836c4b4b669e8dab2e911cf9a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 748617836c4b4b669e8dab2e911cf9a2, State: Initialized, Role: FOLLOWER
I20260925 15:24:22.089414 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 23051
I20260925 15:24:22.089520 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/master-0/wal/instance
I20260925 15:24:22.089334 23119 consensus_queue.cc:260] T 00000000000000000000000000000000 P 748617836c4b4b669e8dab2e911cf9a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "748617836c4b4b669e8dab2e911cf9a2" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 41759 } }
I20260925 15:24:22.089565 23119 raft_consensus.cc:388] T 00000000000000000000000000000000 P 748617836c4b4b669e8dab2e911cf9a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:24:22.089653 23119 raft_consensus.cc:482] T 00000000000000000000000000000000 P 748617836c4b4b669e8dab2e911cf9a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:24:22.089742 3919 external_mini_cluster.cc:1428] Setting key 4ff8fd77fefe46f1ae9328975d91f9ce
I20260925 15:24:22.089756 23119 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 748617836c4b4b669e8dab2e911cf9a2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:22.090720 23119 raft_consensus.cc:504] T 00000000000000000000000000000000 P 748617836c4b4b669e8dab2e911cf9a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "748617836c4b4b669e8dab2e911cf9a2" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 41759 } }
I20260925 15:24:22.090866 23119 leader_election.cc:302] T 00000000000000000000000000000000 P 748617836c4b4b669e8dab2e911cf9a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 748617836c4b4b669e8dab2e911cf9a2; no voters:
I20260925 15:24:22.091076 23119 leader_election.cc:288] T 00000000000000000000000000000000 P 748617836c4b4b669e8dab2e911cf9a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:24:22.091164 23124 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 748617836c4b4b669e8dab2e911cf9a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:24:22.091343 23124 raft_consensus.cc:686] T 00000000000000000000000000000000 P 748617836c4b4b669e8dab2e911cf9a2 [term 1 LEADER]: Becoming Leader. State: Replica: 748617836c4b4b669e8dab2e911cf9a2, State: Running, Role: LEADER
I20260925 15:24:22.091352 23119 sys_catalog.cc:564] T 00000000000000000000000000000000 P 748617836c4b4b669e8dab2e911cf9a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:24:22.091492 23124 consensus_queue.cc:237] T 00000000000000000000000000000000 P 748617836c4b4b669e8dab2e911cf9a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "748617836c4b4b669e8dab2e911cf9a2" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 41759 } }
I20260925 15:24:22.091846 23126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 748617836c4b4b669e8dab2e911cf9a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "748617836c4b4b669e8dab2e911cf9a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "748617836c4b4b669e8dab2e911cf9a2" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 41759 } } }
I20260925 15:24:22.091835 23124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 748617836c4b4b669e8dab2e911cf9a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 748617836c4b4b669e8dab2e911cf9a2. Latest consensus state: current_term: 1 leader_uuid: "748617836c4b4b669e8dab2e911cf9a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "748617836c4b4b669e8dab2e911cf9a2" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 41759 } } }
I20260925 15:24:22.092029 23124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 748617836c4b4b669e8dab2e911cf9a2 [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:22.092228 23126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 748617836c4b4b669e8dab2e911cf9a2 [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:22.092531 23133 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:24:22.093086 23133 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:24:22.094796 23133 catalog_manager.cc:1347] Generated new cluster ID: 94ef18748666450ea12507195d5de26f
I20260925 15:24:22.094864 23133 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 15:24:22 dist-test-slave-4bk3 krb5kdc[23038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:24:22.102420 23123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:22.090386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50552 (local address 127.3.211.254:41759)
0925 15:24:22.091036 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:22.091047 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:22.091087 (+ 40us) server_negotiation.cc:407] Connection header received
0925 15:24:22.091237 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:22.091245 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:22.091307 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:22.091460 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:22.092450 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:22.094063 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:22.095563 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:22.096027 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:22.096235 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:22.098902 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:22.098907 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:22.098908 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:22.098923 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:22.100815 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:22.101192 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:22.101197 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:22.101197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:22.101298 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:22.101512 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:22.101513 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:22.101513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:22.101766 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:22.101956 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:22.102069 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 15:24:22.102215 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":513,"threads_started":1}
I20260925 15:24:22.112128 23133 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:24:22.112749 23133 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:24:22.114743 23133 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 748617836c4b4b669e8dab2e911cf9a2: Generated new TSK 0
I20260925 15:24:22.114970 23133 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.211.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab.
Entry for principal kudu/127.3.211.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab.
I20260925 15:24:22.148743 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.193:0
--local_ip_for_outbound_sockets=127.3.211.193
--webserver_interface=127.3.211.193
--webserver_port=0
--tserver_master_addrs=127.3.211.254:41759
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab
--principal=kudu/127.3.211.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:37007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:24:22.248427 23145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:22.248729 23145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:22.248776 23145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:22.250135 23145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:22.250237 23145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:22.250288 23145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.193
I20260925 15:24:22.252228 23145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:37007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.193.keytab
--principal=kudu/127.3.211.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.211.193
--webserver_port=0
--tserver_master_addrs=127.3.211.254:41759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.193
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:24:22.252624 23145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:22.252908 23145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 15:24:22.259230 23145 server_base.cc:1034] running on GCE node
W20260925 15:24:22.259168 23151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:22.259395 23153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:22.259168 23150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:22.260190 23145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:22.260583 23145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:22.261837 23145 hybrid_clock.cc:648] HybridClock initialized: now 1790349862261782 us; error 63 us; skew 500 ppm
Sep 25 15:24:22 dist-test-slave-4bk3 krb5kdc[23038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349862, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:24:22.266396 23145 init.cc:377] Logged in from keytab as kudu/127.3.211.193@KRBTEST.COM (short username kudu)
I20260925 15:24:22.267043 23145 webserver.cc:460] Webserver started at http://127.3.211.193:43955/ using document root <none> and password file <none>
I20260925 15:24:22.267338 23145 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:22.267405 23145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:22.267544 23145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:22.269014 23145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/ts-0/data/instance:
uuid: "b412e9f9b4024bb893a6d3c01a515c53"
format_stamp: "Formatted at 2026-09-25 15:24:22 on dist-test-slave-4bk3"
server_key: "2c8da19176dbe4cc0a99aba89e44485b"
server_key_iv: "105416f9ea6dedbe865bce91d52a60a2"
server_key_version: "encryptionkey@0"
I20260925 15:24:22.269443 23145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/ts-0/wal/instance:
uuid: "b412e9f9b4024bb893a6d3c01a515c53"
format_stamp: "Formatted at 2026-09-25 15:24:22 on dist-test-slave-4bk3"
server_key: "2c8da19176dbe4cc0a99aba89e44485b"
server_key_iv: "105416f9ea6dedbe865bce91d52a60a2"
server_key_version: "encryptionkey@0"
I20260925 15:24:22.271661 23145 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:24:22.273087 23160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:22.273528 23145 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:24:22.273686 23145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/ts-0/wal
uuid: "b412e9f9b4024bb893a6d3c01a515c53"
format_stamp: "Formatted at 2026-09-25 15:24:22 on dist-test-slave-4bk3"
server_key: "2c8da19176dbe4cc0a99aba89e44485b"
server_key_iv: "105416f9ea6dedbe865bce91d52a60a2"
server_key_version: "encryptionkey@0"
I20260925 15:24:22.273815 23145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:22.285763 23145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:22.288455 23145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:22.288647 23145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:22.288944 23145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:24:22.289307 23145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:24:22.289366 23145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:22.289402 23145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:24:22.289445 23145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:22.297037 23145 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.193:46075
I20260925 15:24:22.297293 23273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.193:46075 every 8 connection(s)
I20260925 15:24:22.297768 23145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:24:22.305353 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 23145
I20260925 15:24:22.305508 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/ts-0/wal/instance
I20260925 15:24:22.305735 3919 external_mini_cluster.cc:1428] Setting key 06a78bbb5cf1cee620b38182b46e6271
Sep 25 15:24:22 dist-test-slave-4bk3 krb5kdc[23038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349862, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.193@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:24:22.311906 23123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:22.298784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:58365 (local address 127.3.211.254:41759)
0925 15:24:22.298984 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:22.298988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:22.299277 (+ 289us) server_negotiation.cc:407] Connection header received
0925 15:24:22.299999 (+ 722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:22.300001 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:22.300024 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:22.300268 (+ 244us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:22.301932 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:22.302531 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:22.303990 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:22.304223 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:22.304317 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:22.307956 (+ 3639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:22.307959 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:22.307959 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:22.307980 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:22.310217 (+ 2237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:22.310795 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:22.310797 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:22.310797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:22.310869 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:22.311150 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:22.311151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:22.311152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:22.311289 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:22.311374 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:22.311700 (+ 326us) server_negotiation.cc:299] Negotiation successful
0925 15:24:22.311799 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260925 15:24:22.312124 23276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:22.298696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:41759 (local address 127.3.211.193:58365)
0925 15:24:22.299035 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:22.299138 (+ 103us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:22.299857 (+ 719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:22.300501 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:22.300507 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:22.301329 (+ 822us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:24:22.301650 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:22.301659 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:22.302689 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:22.302793 (+ 104us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:22.303774 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:22.303782 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:22.304392 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:22.304393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:22.304637 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:22.305355 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:24:22.305365 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:24:22.307780 (+ 2415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:24:22.310371 (+ 2591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:22.310386 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:22.310390 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:22.310687 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:22.310943 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:22.310944 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:22.310945 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:22.311077 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:22.311390 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:24:22.311397 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:24:22.311552 (+ 155us) client_negotiation.cc:769] Sending connection context
0925 15:24:22.311666 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 15:24:22.311861 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":98,"threads_started":1}
I20260925 15:24:22.312697 23274 heartbeater.cc:347] Connected to a master server at 127.3.211.254:41759
I20260925 15:24:22.312801 23274 heartbeater.cc:464] Registering TS with master...
I20260925 15:24:22.313024 23274 heartbeater.cc:511] Master 127.3.211.254:41759 requested a full tablet report, sending...
I20260925 15:24:22.313582 23083 ts_manager.cc:194] Registered new tserver with Master: b412e9f9b4024bb893a6d3c01a515c53 (127.3.211.193:46075)
I20260925 15:24:22.314229 23083 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.193@KRBTEST.COM'} at 127.3.211.193:58365
WARNING: no policy specified for kudu/127.3.211.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab.
Entry for principal kudu/127.3.211.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab.
I20260925 15:24:22.337371 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.194:0
--local_ip_for_outbound_sockets=127.3.211.194
--webserver_interface=127.3.211.194
--webserver_port=0
--tserver_master_addrs=127.3.211.254:41759
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab
--principal=kudu/127.3.211.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:37007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:24:22.433358 23279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:22.433606 23279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:22.433635 23279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:22.435447 23279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:22.435575 23279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:22.435619 23279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.194
I20260925 15:24:22.438395 23279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:37007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.194.keytab
--principal=kudu/127.3.211.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.211.194
--webserver_port=0
--tserver_master_addrs=127.3.211.254:41759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.194
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:24:22.438737 23279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:22.438956 23279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:22.445823 23287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:22.445816 23284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:22.445823 23285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:22.446201 23279 server_base.cc:1034] running on GCE node
I20260925 15:24:22.446472 23279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:22.446743 23279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:22.447959 23279 hybrid_clock.cc:648] HybridClock initialized: now 1790349862447944 us; error 30 us; skew 500 ppm
Sep 25 15:24:22 dist-test-slave-4bk3 krb5kdc[23038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349862, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:24:22.450924 23279 init.cc:377] Logged in from keytab as kudu/127.3.211.194@KRBTEST.COM (short username kudu)
I20260925 15:24:22.451460 23279 webserver.cc:460] Webserver started at http://127.3.211.194:34981/ using document root <none> and password file <none>
I20260925 15:24:22.451723 23279 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:22.451773 23279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:22.451910 23279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:22.453212 23279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/ts-1/data/instance:
uuid: "66c68d9d95ee45c5b46f2aebbd666746"
format_stamp: "Formatted at 2026-09-25 15:24:22 on dist-test-slave-4bk3"
server_key: "2c9f2ca9b9e8db32ac1ef91f7b8f3054"
server_key_iv: "48bb6ded8b20daa66125c7156b05d8d0"
server_key_version: "encryptionkey@0"
I20260925 15:24:22.453637 23279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/ts-1/wal/instance:
uuid: "66c68d9d95ee45c5b46f2aebbd666746"
format_stamp: "Formatted at 2026-09-25 15:24:22 on dist-test-slave-4bk3"
server_key: "2c9f2ca9b9e8db32ac1ef91f7b8f3054"
server_key_iv: "48bb6ded8b20daa66125c7156b05d8d0"
server_key_version: "encryptionkey@0"
I20260925 15:24:22.455348 23279 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:24:22.456532 23294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:22.456750 23279 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:24:22.456842 23279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/ts-1/wal
uuid: "66c68d9d95ee45c5b46f2aebbd666746"
format_stamp: "Formatted at 2026-09-25 15:24:22 on dist-test-slave-4bk3"
server_key: "2c9f2ca9b9e8db32ac1ef91f7b8f3054"
server_key_iv: "48bb6ded8b20daa66125c7156b05d8d0"
server_key_version: "encryptionkey@0"
I20260925 15:24:22.456902 23279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:22.465795 23279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:22.468286 23279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:22.468519 23279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:22.468765 23279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:24:22.469120 23279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:24:22.469182 23279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:22.469229 23279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:24:22.469257 23279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:22.475943 23279 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.194:40693
I20260925 15:24:22.476014 23407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.194:40693 every 8 connection(s)
I20260925 15:24:22.476508 23279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
I20260925 15:24:22.484429 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 23279
I20260925 15:24:22.484555 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/ts-1/wal/instance
Sep 25 15:24:22 dist-test-slave-4bk3 krb5kdc[23038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349862, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.194@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:24:22.484784 3919 external_mini_cluster.cc:1428] Setting key 06b5068393c2f1188634d33551a51a7e
I20260925 15:24:22.490320 23123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:22.477494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:51545 (local address 127.3.211.254:41759)
0925 15:24:22.477614 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:22.477616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:22.477709 (+ 93us) server_negotiation.cc:407] Connection header received
0925 15:24:22.478289 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:22.478291 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:22.478302 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:22.478360 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:22.479786 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:22.480318 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:22.481884 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:22.482102 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:22.482187 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:22.485676 (+ 3489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:22.485677 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:22.485678 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:22.485687 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:22.487780 (+ 2093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:22.488825 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:22.488827 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:22.488827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:22.488919 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:22.489372 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:22.489375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:22.489375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:22.489560 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:22.489664 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:22.490057 (+ 393us) server_negotiation.cc:299] Negotiation successful
0925 15:24:22.490191 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 15:24:22.490576 23410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:22.477398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:41759 (local address 127.3.211.194:51545)
0925 15:24:22.477573 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:22.477622 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:22.478140 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:22.478429 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:22.478434 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:22.479092 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:24:22.479592 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:22.479606 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:22.480550 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:22.480551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:22.481708 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:22.481712 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:22.482256 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:22.482259 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:22.482442 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:22.483176 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:24:22.483186 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:24:22.485514 (+ 2328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:24:22.487964 (+ 2450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:22.487987 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:22.487991 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:22.488595 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:22.489043 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:22.489044 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:22.489045 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:22.489249 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:22.489649 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:24:22.489660 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:24:22.489894 (+ 234us) client_negotiation.cc:769] Sending connection context
0925 15:24:22.490032 (+ 138us) client_negotiation.cc:240] Negotiation successful
0925 15:24:22.490272 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":83,"threads_started":1}
I20260925 15:24:22.491452 23408 heartbeater.cc:347] Connected to a master server at 127.3.211.254:41759
I20260925 15:24:22.491654 23408 heartbeater.cc:464] Registering TS with master...
I20260925 15:24:22.492017 23408 heartbeater.cc:511] Master 127.3.211.254:41759 requested a full tablet report, sending...
I20260925 15:24:22.492942 23083 ts_manager.cc:194] Registered new tserver with Master: 66c68d9d95ee45c5b46f2aebbd666746 (127.3.211.194:40693)
I20260925 15:24:22.493587 23083 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.194@KRBTEST.COM'} at 127.3.211.194:51545
WARNING: no policy specified for kudu/127.3.211.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab.
Entry for principal kudu/127.3.211.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab.
I20260925 15:24:22.520282 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.195:0
--local_ip_for_outbound_sockets=127.3.211.195
--webserver_interface=127.3.211.195
--webserver_port=0
--tserver_master_addrs=127.3.211.254:41759
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab
--principal=kudu/127.3.211.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:37007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:24:22.614463 23413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:22.614800 23413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:22.614848 23413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:22.616182 23413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:22.616233 23413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:22.616264 23413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.195
I20260925 15:24:22.617863 23413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:37007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.195.keytab
--principal=kudu/127.3.211.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.211.195
--webserver_port=0
--tserver_master_addrs=127.3.211.254:41759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.195
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:24:22.618080 23413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:22.618274 23413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:22.624580 23421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:22.624639 23418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:22.624686 23413 server_base.cc:1034] running on GCE node
W20260925 15:24:22.624657 23419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:22.625013 23413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:22.625303 23413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:22.626511 23413 hybrid_clock.cc:648] HybridClock initialized: now 1790349862626495 us; error 33 us; skew 500 ppm
Sep 25 15:24:22 dist-test-slave-4bk3 krb5kdc[23038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349862, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:24:22.630277 23413 init.cc:377] Logged in from keytab as kudu/127.3.211.195@KRBTEST.COM (short username kudu)
I20260925 15:24:22.630923 23413 webserver.cc:460] Webserver started at http://127.3.211.195:37201/ using document root <none> and password file <none>
I20260925 15:24:22.631209 23413 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:22.631289 23413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:22.631451 23413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:22.632738 23413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/ts-2/data/instance:
uuid: "3efbceb6725b4a14a2f495fc2239d1d5"
format_stamp: "Formatted at 2026-09-25 15:24:22 on dist-test-slave-4bk3"
server_key: "2d4859bb09f4e73fb2ed95b01e4bebf4"
server_key_iv: "1afabb746e4530f20169c1bd7089ccca"
server_key_version: "encryptionkey@0"
I20260925 15:24:22.633152 23413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/ts-2/wal/instance:
uuid: "3efbceb6725b4a14a2f495fc2239d1d5"
format_stamp: "Formatted at 2026-09-25 15:24:22 on dist-test-slave-4bk3"
server_key: "2d4859bb09f4e73fb2ed95b01e4bebf4"
server_key_iv: "1afabb746e4530f20169c1bd7089ccca"
server_key_version: "encryptionkey@0"
I20260925 15:24:22.634812 23413 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:24:22.636121 23428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:22.636475 23413 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:24:22.636598 23413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/ts-2/wal
uuid: "3efbceb6725b4a14a2f495fc2239d1d5"
format_stamp: "Formatted at 2026-09-25 15:24:22 on dist-test-slave-4bk3"
server_key: "2d4859bb09f4e73fb2ed95b01e4bebf4"
server_key_iv: "1afabb746e4530f20169c1bd7089ccca"
server_key_version: "encryptionkey@0"
I20260925 15:24:22.636698 23413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:22.645177 23413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:22.648540 23413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:22.648731 23413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:22.649034 23413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:24:22.649560 23413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:24:22.649663 23413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:22.649752 23413 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:24:22.649775 23413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:22.657341 23413 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.195:41297
I20260925 15:24:22.657437 23541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.195:41297 every 8 connection(s)
I20260925 15:24:22.657802 23413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
Sep 25 15:24:22 dist-test-slave-4bk3 krb5kdc[23038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349862, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.195@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:24:22.666616 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 23413
I20260925 15:24:22.666759 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/ts-2/wal/instance
I20260925 15:24:22.666991 3919 external_mini_cluster.cc:1428] Setting key 0762739123decd1598c7bf9a3461c1de
I20260925 15:24:22.671264 23123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:22.658636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:48323 (local address 127.3.211.254:41759)
0925 15:24:22.658719 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:22.658721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:22.658953 (+ 232us) server_negotiation.cc:407] Connection header received
0925 15:24:22.659472 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:22.659473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:22.659485 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:22.659560 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:22.661185 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:22.661725 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:22.663092 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:22.663288 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:22.663365 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:22.666721 (+ 3356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:22.666723 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:22.666724 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:22.666737 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:22.668816 (+ 2079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:22.669748 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:22.669750 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:22.669750 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:22.669828 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:22.670315 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:22.670316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:22.670317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:22.670469 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:22.670633 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:22.671092 (+ 459us) server_negotiation.cc:299] Negotiation successful
0925 15:24:22.671187 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 15:24:22.671458 23544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:22.658602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:41759 (local address 127.3.211.195:48323)
0925 15:24:22.658759 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:22.658806 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:22.659308 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:22.659854 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:22.659862 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:22.660737 (+ 875us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:24:22.661012 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:22.661018 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:22.661883 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:22.661884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:22.662954 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:22.662958 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:22.663408 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:22.663409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:22.663537 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:22.664251 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:24:22.664261 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:24:22.666568 (+ 2307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:24:22.669057 (+ 2489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:22.669139 (+ 82us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:22.669143 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:22.669600 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:22.669954 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:22.669955 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:22.669956 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:22.670124 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:22.670677 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:24:22.670690 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:24:22.670891 (+ 201us) client_negotiation.cc:769] Sending connection context
0925 15:24:22.671037 (+ 146us) client_negotiation.cc:240] Negotiation successful
0925 15:24:22.671207 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":88,"threads_started":1}
I20260925 15:24:22.672286 23542 heartbeater.cc:347] Connected to a master server at 127.3.211.254:41759
I20260925 15:24:22.672492 23542 heartbeater.cc:464] Registering TS with master...
I20260925 15:24:22.672734 23542 heartbeater.cc:511] Master 127.3.211.254:41759 requested a full tablet report, sending...
I20260925 15:24:22.673244 23083 ts_manager.cc:194] Registered new tserver with Master: 3efbceb6725b4a14a2f495fc2239d1d5 (127.3.211.195:41297)
I20260925 15:24:22.673662 23083 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.211.195@KRBTEST.COM'} at 127.3.211.195:48323
I20260925 15:24:22.680094 3919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:24:22.690380 23123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:22.681695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46018 (local address 127.3.211.254:41759)
0925 15:24:22.681825 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:22.681827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:22.681897 (+ 70us) server_negotiation.cc:407] Connection header received
0925 15:24:22.681968 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:22.681969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:22.681983 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:22.682203 (+ 220us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:22.683132 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:22.683671 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:22.684795 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:22.685073 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:22.685185 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:22.686842 (+ 1657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:22.686844 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:22.686845 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:22.686854 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:22.688813 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:22.689315 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:22.689317 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:22.689318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:22.689413 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:22.689678 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:22.689679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:22.689679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:22.689879 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:22.689985 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:22.690200 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 15:24:22.690306 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 15:24:22.693275 23083 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46018:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:24:22.695124 23083 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:24:22.702797 23558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:22.698339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59676 (local address 127.3.211.195:41297)
0925 15:24:22.698520 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:22.698524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:22.698582 (+ 58us) server_negotiation.cc:407] Connection header received
0925 15:24:22.698868 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:22.698871 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:22.698887 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:22.698959 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:24:22.699191 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:22.699896 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:22.701541 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:22.702205 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:22.702276 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:22.702325 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:22.702630 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 15:24:22.702681 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":80,"threads_started":1}
I20260925 15:24:22.702782 23554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:22.697951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.195:41297 (local address 127.0.0.1:59676)
0925 15:24:22.698526 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:22.698536 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:22.698597 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:22.699046 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:22.699047 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:22.699049 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:24:22.699102 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:22.699103 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:22.700020 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:22.700021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:22.701410 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:22.701415 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:22.702314 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:22.702316 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:22.702438 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:22.702462 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 15:24:22.702573 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 15:24:22.702665 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":93,"threads_started":1}
I20260925 15:24:22.703074 23557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:22.698243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50548 (local address 127.3.211.193:46075)
0925 15:24:22.698445 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:22.698449 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:22.698528 (+ 79us) server_negotiation.cc:407] Connection header received
0925 15:24:22.698660 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:22.698663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:22.698679 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:22.698764 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:24:22.698904 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:22.700216 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:22.701778 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:22.702512 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:22.702605 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:22.702675 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:22.702885 (+ 210us) server_negotiation.cc:299] Negotiation successful
0925 15:24:22.702980 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":88,"threads_started":1}
I20260925 15:24:22.703565 23555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:22.697938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:46075 (local address 127.0.0.1:50548)
0925 15:24:22.698415 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:22.698462 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:22.698558 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:22.698774 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:22.698775 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:22.698777 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:24:22.698814 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:22.698815 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:22.700445 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:22.700446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:22.701636 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:22.701640 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:22.702641 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:22.702642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:22.702727 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:22.702735 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 15:24:22.703368 (+ 633us) client_negotiation.cc:240] Negotiation successful
0925 15:24:22.703446 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"mutex_wait_us":3,"thread_start_us":101,"threads_started":1}
I20260925 15:24:22.703897 23476 tablet_service.cc:1467] Processing CreateTablet for tablet ed20879700a24404890df459136bea51 (DEFAULT_TABLE table=test-table [id=751a04c37f264ddfa1690e5b1fe08127]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:24:22.704396 23553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:22.697923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.194:40693 (local address 127.0.0.1:48662)
0925 15:24:22.698040 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:22.698080 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:22.698193 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:22.698654 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:22.698657 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:22.698659 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:24:22.698776 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:22.698777 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:22.700437 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:22.700438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:22.701656 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:22.701659 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:22.704038 (+ 2379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:22.704039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:22.704129 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:22.704134 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:24:22.704229 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 15:24:22.704282 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"mutex_wait_us":34,"thread_start_us":63,"threads_started":1}
I20260925 15:24:22.704487 23556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:22.698239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48662 (local address 127.3.211.194:40693)
0925 15:24:22.698399 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:22.698405 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:22.698417 (+ 12us) server_negotiation.cc:407] Connection header received
0925 15:24:22.698462 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:22.698464 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:22.698483 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:22.698559 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:24:22.698862 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:22.700162 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:22.702734 (+ 2572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:22.703934 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:22.704026 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:22.704087 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:22.704273 (+ 186us) server_negotiation.cc:299] Negotiation successful
0925 15:24:22.704366 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":90,"threads_started":1}
I20260925 15:24:22.704516 23476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed20879700a24404890df459136bea51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:22.704974 23208 tablet_service.cc:1467] Processing CreateTablet for tablet ed20879700a24404890df459136bea51 (DEFAULT_TABLE table=test-table [id=751a04c37f264ddfa1690e5b1fe08127]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:24:22.705246 23341 tablet_service.cc:1467] Processing CreateTablet for tablet ed20879700a24404890df459136bea51 (DEFAULT_TABLE table=test-table [id=751a04c37f264ddfa1690e5b1fe08127]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:24:22.705420 23208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed20879700a24404890df459136bea51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:22.705543 23341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed20879700a24404890df459136bea51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:22.708276 23559 tablet_bootstrap.cc:492] T ed20879700a24404890df459136bea51 P 3efbceb6725b4a14a2f495fc2239d1d5: Bootstrap starting.
I20260925 15:24:22.709139 23559 tablet_bootstrap.cc:654] T ed20879700a24404890df459136bea51 P 3efbceb6725b4a14a2f495fc2239d1d5: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:22.709635 23559 log.cc:824] T ed20879700a24404890df459136bea51 P 3efbceb6725b4a14a2f495fc2239d1d5: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:22.710412 23561 tablet_bootstrap.cc:492] T ed20879700a24404890df459136bea51 P 66c68d9d95ee45c5b46f2aebbd666746: Bootstrap starting.
I20260925 15:24:22.710923 23559 tablet_bootstrap.cc:492] T ed20879700a24404890df459136bea51 P 3efbceb6725b4a14a2f495fc2239d1d5: No bootstrap required, opened a new log
I20260925 15:24:22.711045 23559 ts_tablet_manager.cc:1397] T ed20879700a24404890df459136bea51 P 3efbceb6725b4a14a2f495fc2239d1d5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 15:24:22.711484 23562 tablet_bootstrap.cc:492] T ed20879700a24404890df459136bea51 P b412e9f9b4024bb893a6d3c01a515c53: Bootstrap starting.
I20260925 15:24:22.711671 23561 tablet_bootstrap.cc:654] T ed20879700a24404890df459136bea51 P 66c68d9d95ee45c5b46f2aebbd666746: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:22.712450 23561 log.cc:824] T ed20879700a24404890df459136bea51 P 66c68d9d95ee45c5b46f2aebbd666746: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:22.712493 23562 tablet_bootstrap.cc:654] T ed20879700a24404890df459136bea51 P b412e9f9b4024bb893a6d3c01a515c53: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:22.712738 23559 raft_consensus.cc:348] T ed20879700a24404890df459136bea51 P 3efbceb6725b4a14a2f495fc2239d1d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66c68d9d95ee45c5b46f2aebbd666746" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40693 } } peers { permanent_uuid: "b412e9f9b4024bb893a6d3c01a515c53" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 46075 } } peers { permanent_uuid: "3efbceb6725b4a14a2f495fc2239d1d5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41297 } }
I20260925 15:24:22.712997 23559 raft_consensus.cc:374] T ed20879700a24404890df459136bea51 P 3efbceb6725b4a14a2f495fc2239d1d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:24:22.713042 23559 raft_consensus.cc:729] T ed20879700a24404890df459136bea51 P 3efbceb6725b4a14a2f495fc2239d1d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efbceb6725b4a14a2f495fc2239d1d5, State: Initialized, Role: FOLLOWER
I20260925 15:24:22.713151 23562 log.cc:824] T ed20879700a24404890df459136bea51 P b412e9f9b4024bb893a6d3c01a515c53: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:22.713222 23559 consensus_queue.cc:260] T ed20879700a24404890df459136bea51 P 3efbceb6725b4a14a2f495fc2239d1d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66c68d9d95ee45c5b46f2aebbd666746" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40693 } } peers { permanent_uuid: "b412e9f9b4024bb893a6d3c01a515c53" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 46075 } } peers { permanent_uuid: "3efbceb6725b4a14a2f495fc2239d1d5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41297 } }
I20260925 15:24:22.713471 23561 tablet_bootstrap.cc:492] T ed20879700a24404890df459136bea51 P 66c68d9d95ee45c5b46f2aebbd666746: No bootstrap required, opened a new log
I20260925 15:24:22.713560 23561 ts_tablet_manager.cc:1397] T ed20879700a24404890df459136bea51 P 66c68d9d95ee45c5b46f2aebbd666746: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260925 15:24:22.713558 23559 ts_tablet_manager.cc:1428] T ed20879700a24404890df459136bea51 P 3efbceb6725b4a14a2f495fc2239d1d5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 15:24:22.713590 23542 heartbeater.cc:503] Master 127.3.211.254:41759 was elected leader, sending a full tablet report...
I20260925 15:24:22.714093 23562 tablet_bootstrap.cc:492] T ed20879700a24404890df459136bea51 P b412e9f9b4024bb893a6d3c01a515c53: No bootstrap required, opened a new log
I20260925 15:24:22.714178 23562 ts_tablet_manager.cc:1397] T ed20879700a24404890df459136bea51 P b412e9f9b4024bb893a6d3c01a515c53: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260925 15:24:22.715780 23561 raft_consensus.cc:348] T ed20879700a24404890df459136bea51 P 66c68d9d95ee45c5b46f2aebbd666746 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66c68d9d95ee45c5b46f2aebbd666746" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40693 } } peers { permanent_uuid: "b412e9f9b4024bb893a6d3c01a515c53" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 46075 } } peers { permanent_uuid: "3efbceb6725b4a14a2f495fc2239d1d5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41297 } }
I20260925 15:24:22.716033 23561 raft_consensus.cc:374] T ed20879700a24404890df459136bea51 P 66c68d9d95ee45c5b46f2aebbd666746 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:24:22.716082 23561 raft_consensus.cc:729] T ed20879700a24404890df459136bea51 P 66c68d9d95ee45c5b46f2aebbd666746 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66c68d9d95ee45c5b46f2aebbd666746, State: Initialized, Role: FOLLOWER
I20260925 15:24:22.716210 23561 consensus_queue.cc:260] T ed20879700a24404890df459136bea51 P 66c68d9d95ee45c5b46f2aebbd666746 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66c68d9d95ee45c5b46f2aebbd666746" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40693 } } peers { permanent_uuid: "b412e9f9b4024bb893a6d3c01a515c53" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 46075 } } peers { permanent_uuid: "3efbceb6725b4a14a2f495fc2239d1d5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41297 } }
I20260925 15:24:22.716388 23562 raft_consensus.cc:348] T ed20879700a24404890df459136bea51 P b412e9f9b4024bb893a6d3c01a515c53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66c68d9d95ee45c5b46f2aebbd666746" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40693 } } peers { permanent_uuid: "b412e9f9b4024bb893a6d3c01a515c53" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 46075 } } peers { permanent_uuid: "3efbceb6725b4a14a2f495fc2239d1d5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41297 } }
I20260925 15:24:22.716578 23561 ts_tablet_manager.cc:1428] T ed20879700a24404890df459136bea51 P 66c68d9d95ee45c5b46f2aebbd666746: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:24:22.716589 23562 raft_consensus.cc:374] T ed20879700a24404890df459136bea51 P b412e9f9b4024bb893a6d3c01a515c53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:24:22.716625 23562 raft_consensus.cc:729] T ed20879700a24404890df459136bea51 P b412e9f9b4024bb893a6d3c01a515c53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b412e9f9b4024bb893a6d3c01a515c53, State: Initialized, Role: FOLLOWER
I20260925 15:24:22.716776 23408 heartbeater.cc:503] Master 127.3.211.254:41759 was elected leader, sending a full tablet report...
I20260925 15:24:22.716749 23562 consensus_queue.cc:260] T ed20879700a24404890df459136bea51 P b412e9f9b4024bb893a6d3c01a515c53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66c68d9d95ee45c5b46f2aebbd666746" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40693 } } peers { permanent_uuid: "b412e9f9b4024bb893a6d3c01a515c53" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 46075 } } peers { permanent_uuid: "3efbceb6725b4a14a2f495fc2239d1d5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41297 } }
I20260925 15:24:22.717049 23562 ts_tablet_manager.cc:1428] T ed20879700a24404890df459136bea51 P b412e9f9b4024bb893a6d3c01a515c53: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260925 15:24:22.717139 23274 heartbeater.cc:503] Master 127.3.211.254:41759 was elected leader, sending a full tablet report...
W20260925 15:24:22.727254 23409 tablet.cc:2369] T ed20879700a24404890df459136bea51 P 66c68d9d95ee45c5b46f2aebbd666746: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:24:22.765837 23565 raft_consensus.cc:482] T ed20879700a24404890df459136bea51 P 3efbceb6725b4a14a2f495fc2239d1d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:24:22.766029 23565 raft_consensus.cc:504] T ed20879700a24404890df459136bea51 P 3efbceb6725b4a14a2f495fc2239d1d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66c68d9d95ee45c5b46f2aebbd666746" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40693 } } peers { permanent_uuid: "b412e9f9b4024bb893a6d3c01a515c53" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 46075 } } peers { permanent_uuid: "3efbceb6725b4a14a2f495fc2239d1d5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41297 } }
I20260925 15:24:22.766459 23565 leader_election.cc:288] T ed20879700a24404890df459136bea51 P 3efbceb6725b4a14a2f495fc2239d1d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66c68d9d95ee45c5b46f2aebbd666746 (127.3.211.194:40693), b412e9f9b4024bb893a6d3c01a515c53 (127.3.211.193:46075)
I20260925 15:24:22.769819 23568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:22.766696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.194:40693 (local address 127.3.211.195:55835)
0925 15:24:22.766983 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:22.766992 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:22.767071 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:22.767228 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:22.767229 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:22.767231 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:24:22.767306 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:22.767307 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:22.767955 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:22.767956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:22.768773 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:22.768775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:22.769496 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:22.769498 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:22.769589 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:22.769594 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:24:22.769670 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 15:24:22.769719 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":82,"thread_start_us":129,"threads_started":1}
I20260925 15:24:22.769801 23556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:22.766782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:55835 (local address 127.3.211.194:40693)
0925 15:24:22.766855 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:22.766857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:22.767063 (+ 206us) server_negotiation.cc:407] Connection header received
0925 15:24:22.767143 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:22.767144 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:22.767153 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:22.767225 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:24:22.767385 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:22.767864 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:22.768883 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:22.769275 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:22.769501 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:22.769516 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:22.769676 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 15:24:22.769726 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 15:24:22.770006 23544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:22.766687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:46075 (local address 127.3.211.195:50171)
0925 15:24:22.766740 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:22.766753 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:22.766824 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:22.767020 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:22.767021 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:22.767022 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:24:22.767100 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:22.767102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:22.767763 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:22.767764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:22.768554 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:22.768556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:22.769422 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:22.769424 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:22.769566 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:22.769571 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:24:22.769830 (+ 259us) client_negotiation.cc:240] Negotiation successful
0925 15:24:22.769910 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 15:24:22.770006 23557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:22.766839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:50171 (local address 127.3.211.193:46075)
0925 15:24:22.766888 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:22.766890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:22.766898 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:24:22.766931 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:22.766932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:22.766939 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:22.767014 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:24:22.767174 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:22.767665 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:22.768663 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:22.769212 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:22.769408 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:22.769431 (+ 23us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:22.769847 (+ 416us) server_negotiation.cc:299] Negotiation successful
0925 15:24:22.769917 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 15:24:22.770370 23362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed20879700a24404890df459136bea51" candidate_uuid: "3efbceb6725b4a14a2f495fc2239d1d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66c68d9d95ee45c5b46f2aebbd666746" is_pre_election: true
I20260925 15:24:22.770596 23362 raft_consensus.cc:2454] T ed20879700a24404890df459136bea51 P 66c68d9d95ee45c5b46f2aebbd666746 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3efbceb6725b4a14a2f495fc2239d1d5 in term 0.
I20260925 15:24:22.770599 23228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed20879700a24404890df459136bea51" candidate_uuid: "3efbceb6725b4a14a2f495fc2239d1d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b412e9f9b4024bb893a6d3c01a515c53" is_pre_election: true
I20260925 15:24:22.770778 23228 raft_consensus.cc:2454] T ed20879700a24404890df459136bea51 P b412e9f9b4024bb893a6d3c01a515c53 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3efbceb6725b4a14a2f495fc2239d1d5 in term 0.
I20260925 15:24:22.770884 23432 leader_election.cc:302] T ed20879700a24404890df459136bea51 P 3efbceb6725b4a14a2f495fc2239d1d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efbceb6725b4a14a2f495fc2239d1d5, 66c68d9d95ee45c5b46f2aebbd666746; no voters:
I20260925 15:24:22.771046 23565 raft_consensus.cc:2781] T ed20879700a24404890df459136bea51 P 3efbceb6725b4a14a2f495fc2239d1d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:24:22.771106 23565 raft_consensus.cc:482] T ed20879700a24404890df459136bea51 P 3efbceb6725b4a14a2f495fc2239d1d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:24:22.771132 23565 raft_consensus.cc:3037] T ed20879700a24404890df459136bea51 P 3efbceb6725b4a14a2f495fc2239d1d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:22.771865 23565 raft_consensus.cc:504] T ed20879700a24404890df459136bea51 P 3efbceb6725b4a14a2f495fc2239d1d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66c68d9d95ee45c5b46f2aebbd666746" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40693 } } peers { permanent_uuid: "b412e9f9b4024bb893a6d3c01a515c53" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 46075 } } peers { permanent_uuid: "3efbceb6725b4a14a2f495fc2239d1d5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41297 } }
I20260925 15:24:22.772032 23565 leader_election.cc:288] T ed20879700a24404890df459136bea51 P 3efbceb6725b4a14a2f495fc2239d1d5 [CANDIDATE]: Term 1 election: Requested vote from peers 66c68d9d95ee45c5b46f2aebbd666746 (127.3.211.194:40693), b412e9f9b4024bb893a6d3c01a515c53 (127.3.211.193:46075)
I20260925 15:24:22.772276 23362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed20879700a24404890df459136bea51" candidate_uuid: "3efbceb6725b4a14a2f495fc2239d1d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66c68d9d95ee45c5b46f2aebbd666746"
I20260925 15:24:22.772271 23228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed20879700a24404890df459136bea51" candidate_uuid: "3efbceb6725b4a14a2f495fc2239d1d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b412e9f9b4024bb893a6d3c01a515c53"
I20260925 15:24:22.772406 23362 raft_consensus.cc:3037] T ed20879700a24404890df459136bea51 P 66c68d9d95ee45c5b46f2aebbd666746 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:22.772420 23228 raft_consensus.cc:3037] T ed20879700a24404890df459136bea51 P b412e9f9b4024bb893a6d3c01a515c53 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:22.773383 23362 raft_consensus.cc:2454] T ed20879700a24404890df459136bea51 P 66c68d9d95ee45c5b46f2aebbd666746 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3efbceb6725b4a14a2f495fc2239d1d5 in term 1.
I20260925 15:24:22.773384 23228 raft_consensus.cc:2454] T ed20879700a24404890df459136bea51 P b412e9f9b4024bb893a6d3c01a515c53 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3efbceb6725b4a14a2f495fc2239d1d5 in term 1.
I20260925 15:24:22.773609 23432 leader_election.cc:302] T ed20879700a24404890df459136bea51 P 3efbceb6725b4a14a2f495fc2239d1d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efbceb6725b4a14a2f495fc2239d1d5, 66c68d9d95ee45c5b46f2aebbd666746; no voters:
I20260925 15:24:22.773725 23565 raft_consensus.cc:2781] T ed20879700a24404890df459136bea51 P 3efbceb6725b4a14a2f495fc2239d1d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:24:22.773922 23565 raft_consensus.cc:686] T ed20879700a24404890df459136bea51 P 3efbceb6725b4a14a2f495fc2239d1d5 [term 1 LEADER]: Becoming Leader. State: Replica: 3efbceb6725b4a14a2f495fc2239d1d5, State: Running, Role: LEADER
I20260925 15:24:22.774034 23565 consensus_queue.cc:237] T ed20879700a24404890df459136bea51 P 3efbceb6725b4a14a2f495fc2239d1d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66c68d9d95ee45c5b46f2aebbd666746" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40693 } } peers { permanent_uuid: "b412e9f9b4024bb893a6d3c01a515c53" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 46075 } } peers { permanent_uuid: "3efbceb6725b4a14a2f495fc2239d1d5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41297 } }
I20260925 15:24:22.775028 23083 catalog_manager.cc:5495] T ed20879700a24404890df459136bea51 P 3efbceb6725b4a14a2f495fc2239d1d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3efbceb6725b4a14a2f495fc2239d1d5 (127.3.211.195). New cstate: current_term: 1 leader_uuid: "3efbceb6725b4a14a2f495fc2239d1d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66c68d9d95ee45c5b46f2aebbd666746" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b412e9f9b4024bb893a6d3c01a515c53" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 46075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3efbceb6725b4a14a2f495fc2239d1d5" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 41297 } health_report { overall_health: HEALTHY } } }
I20260925 15:24:22.797513 23558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:22.794385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59692 (local address 127.3.211.195:41297)
0925 15:24:22.794452 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:22.794454 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:22.794467 (+ 13us) server_negotiation.cc:407] Connection header received
0925 15:24:22.794510 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:22.794511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:22.794530 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:22.794598 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:24:22.794925 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:22.795439 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:22.796289 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:22.796533 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:22.796598 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:22.796820 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:24:22.796944 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:24:22.797025 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:22.797284 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 15:24:22.797342 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260925 15:24:22.798727 23275 tablet.cc:2369] T ed20879700a24404890df459136bea51 P b412e9f9b4024bb893a6d3c01a515c53: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:24:22.799616 23228 raft_consensus.cc:1261] T ed20879700a24404890df459136bea51 P b412e9f9b4024bb893a6d3c01a515c53 [term 1 FOLLOWER]: Refusing update from remote peer 3efbceb6725b4a14a2f495fc2239d1d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:24:22.799818 23362 raft_consensus.cc:1261] T ed20879700a24404890df459136bea51 P 66c68d9d95ee45c5b46f2aebbd666746 [term 1 FOLLOWER]: Refusing update from remote peer 3efbceb6725b4a14a2f495fc2239d1d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:24:22.800031 23569 consensus_queue.cc:1035] T ed20879700a24404890df459136bea51 P 3efbceb6725b4a14a2f495fc2239d1d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b412e9f9b4024bb893a6d3c01a515c53" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 46075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:24:22.800177 23565 consensus_queue.cc:1035] T ed20879700a24404890df459136bea51 P 3efbceb6725b4a14a2f495fc2239d1d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66c68d9d95ee45c5b46f2aebbd666746" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 40693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:24:22.804270 23574 mvcc.cc:205] Tried to move back new op lower bound from 7333273038024540160 to 7333273037923307520. Current Snapshot: MvccSnapshot[applied={T|T < 7333273038024540160}]
I20260925 15:24:22.806286 23083 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:24:22.806437 23083 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:24:22.807777 23083 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 748617836c4b4b669e8dab2e911cf9a2: Sending DeleteTablet for 3 replicas of tablet ed20879700a24404890df459136bea51
I20260925 15:24:22.808375 23341 tablet_service.cc:1514] Processing DeleteTablet for tablet ed20879700a24404890df459136bea51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:24:22 UTC) from {username='kudu', principal='kudu/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:48662
I20260925 15:24:22.808432 23208 tablet_service.cc:1514] Processing DeleteTablet for tablet ed20879700a24404890df459136bea51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:24:22 UTC) from {username='kudu', principal='kudu/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:50548
I20260925 15:24:22.808621 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 23145
I20260925 15:24:22.808689 23476 tablet_service.cc:1514] Processing DeleteTablet for tablet ed20879700a24404890df459136bea51 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:24:22 UTC) from {username='kudu', principal='kudu/127.3.211.254@KRBTEST.COM'} at 127.0.0.1:59676
I20260925 15:24:22.809260 23581 tablet_replica.cc:330] T ed20879700a24404890df459136bea51 P 66c68d9d95ee45c5b46f2aebbd666746: stopping tablet replica
I20260925 15:24:22.809412 23581 raft_consensus.cc:2229] T ed20879700a24404890df459136bea51 P 66c68d9d95ee45c5b46f2aebbd666746 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:24:22.809561 23581 raft_consensus.cc:2258] T ed20879700a24404890df459136bea51 P 66c68d9d95ee45c5b46f2aebbd666746 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:24:22.810017 23581 ts_tablet_manager.cc:1905] T ed20879700a24404890df459136bea51 P 66c68d9d95ee45c5b46f2aebbd666746: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:24:22.811707 23581 ts_tablet_manager.cc:1918] T ed20879700a24404890df459136bea51 P 66c68d9d95ee45c5b46f2aebbd666746: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:24:22.811841 23581 log.cc:1196] T ed20879700a24404890df459136bea51 P 66c68d9d95ee45c5b46f2aebbd666746: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/ts-1/wal/wals/ed20879700a24404890df459136bea51
I20260925 15:24:22.812147 23581 ts_tablet_manager.cc:1939] T ed20879700a24404890df459136bea51 P 66c68d9d95ee45c5b46f2aebbd666746: Deleting consensus metadata
I20260925 15:24:22.812664 23070 catalog_manager.cc:4841] TS 66c68d9d95ee45c5b46f2aebbd666746 (127.3.211.194:40693): tablet ed20879700a24404890df459136bea51 (table test-table [id=751a04c37f264ddfa1690e5b1fe08127]) successfully deleted
I20260925 15:24:22.816577 23583 tablet_replica.cc:330] T ed20879700a24404890df459136bea51 P 3efbceb6725b4a14a2f495fc2239d1d5: stopping tablet replica
W20260925 15:24:22.816929 23067 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.211.193:46075 (error 108)
I20260925 15:24:22.816954 23583 raft_consensus.cc:2229] T ed20879700a24404890df459136bea51 P 3efbceb6725b4a14a2f495fc2239d1d5 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:24:22.817118 23583 raft_consensus.cc:2258] T ed20879700a24404890df459136bea51 P 3efbceb6725b4a14a2f495fc2239d1d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:24:22.817622 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 23279
I20260925 15:24:22.817672 23583 ts_tablet_manager.cc:1905] T ed20879700a24404890df459136bea51 P 3efbceb6725b4a14a2f495fc2239d1d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:24:22.820369 23583 ts_tablet_manager.cc:1918] T ed20879700a24404890df459136bea51 P 3efbceb6725b4a14a2f495fc2239d1d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:24:22.820467 23583 log.cc:1196] T ed20879700a24404890df459136bea51 P 3efbceb6725b4a14a2f495fc2239d1d5: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790349610595785-3919-0/minicluster-data/ts-2/wal/wals/ed20879700a24404890df459136bea51
I20260925 15:24:22.820765 23583 ts_tablet_manager.cc:1939] T ed20879700a24404890df459136bea51 P 3efbceb6725b4a14a2f495fc2239d1d5: Deleting consensus metadata
I20260925 15:24:22.821334 23068 catalog_manager.cc:4841] TS 3efbceb6725b4a14a2f495fc2239d1d5 (127.3.211.195:41297): tablet ed20879700a24404890df459136bea51 (table test-table [id=751a04c37f264ddfa1690e5b1fe08127]) successfully deleted
I20260925 15:24:22.821597 23554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:22.821463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:46075 (local address 127.0.0.1:50560)
0925 15:24:22.821516 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:22.821553 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.193:46075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260925 15:24:22.821731 23067 catalog_manager.cc:4584] TS b412e9f9b4024bb893a6d3c01a515c53 (127.3.211.193:46075): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ed20879700a24404890df459136bea51: Network error: Client connection negotiation failed: client connection to 127.3.211.193:46075: connect: Connection refused (error 111)
I20260925 15:24:22.823694 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 23413
I20260925 15:24:22.828616 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 23051
2026-09-25T15:24:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2094 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T15:24:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:24:22Z Disabled control of system clock
I20260925 15:24:22.857903 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:38181
--webserver_interface=127.3.211.254
--webserver_port=0
--builtin_ntp_servers=127.3.211.212:37785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.211.254:38181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:24:22.949867 23588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:22.950230 23588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:22.950258 23588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:22.952570 23588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:24:22.952665 23588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:22.952680 23588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:22.952695 23588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:24:22.952710 23588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:24:22.955967 23588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:37785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.211.254:38181
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:38181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.211.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:24:22.956386 23588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:22.956796 23588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:22.962850 23596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:22.962857 23593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:22.963048 23588 server_base.cc:1034] running on GCE node
W20260925 15:24:22.962862 23594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:22.963467 23588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:22.963812 23588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:22.965085 23588 hybrid_clock.cc:648] HybridClock initialized: now 1790349862965051 us; error 37 us; skew 500 ppm
I20260925 15:24:22.967273 23588 webserver.cc:460] Webserver started at http://127.3.211.254:46717/ using document root <none> and password file <none>
I20260925 15:24:22.967634 23588 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:22.967724 23588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:22.967908 23588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:22.969496 23588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/master-0/data/instance:
uuid: "21d3993a917842979c51dc7c09fabd9b"
format_stamp: "Formatted at 2026-09-25 15:24:22 on dist-test-slave-4bk3"
server_key: "bb1959f1f1c7c3f6dbd166421ceb4e02"
server_key_iv: "e6686ff565c03f29bae972027b925762"
server_key_version: "encryptionkey@0"
I20260925 15:24:22.969998 23588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/master-0/wal/instance:
uuid: "21d3993a917842979c51dc7c09fabd9b"
format_stamp: "Formatted at 2026-09-25 15:24:22 on dist-test-slave-4bk3"
server_key: "bb1959f1f1c7c3f6dbd166421ceb4e02"
server_key_iv: "e6686ff565c03f29bae972027b925762"
server_key_version: "encryptionkey@0"
I20260925 15:24:22.971930 23588 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260925 15:24:22.973245 23602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:22.973589 23588 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:24:22.973739 23588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/master-0/wal
uuid: "21d3993a917842979c51dc7c09fabd9b"
format_stamp: "Formatted at 2026-09-25 15:24:22 on dist-test-slave-4bk3"
server_key: "bb1959f1f1c7c3f6dbd166421ceb4e02"
server_key_iv: "e6686ff565c03f29bae972027b925762"
server_key_version: "encryptionkey@0"
I20260925 15:24:22.973873 23588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:22.995728 23588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:22.996146 23588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:22.996316 23588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:23.000566 23588 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.254:38181
I20260925 15:24:23.000625 23654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.254:38181 every 8 connection(s)
I20260925 15:24:23.001020 23588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
I20260925 15:24:23.002192 23655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:23.004732 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 23588
I20260925 15:24:23.004877 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/master-0/wal/instance
I20260925 15:24:23.005091 3919 external_mini_cluster.cc:1428] Setting key 913373dbdbede9dcf1fb4c6836c16428
I20260925 15:24:23.005347 23655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21d3993a917842979c51dc7c09fabd9b: Bootstrap starting.
I20260925 15:24:23.006179 23655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21d3993a917842979c51dc7c09fabd9b: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:23.006557 23655 log.cc:824] T 00000000000000000000000000000000 P 21d3993a917842979c51dc7c09fabd9b: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:23.007951 23655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21d3993a917842979c51dc7c09fabd9b: No bootstrap required, opened a new log
I20260925 15:24:23.011355 23655 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21d3993a917842979c51dc7c09fabd9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21d3993a917842979c51dc7c09fabd9b" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38181 } }
I20260925 15:24:23.011627 23655 raft_consensus.cc:374] T 00000000000000000000000000000000 P 21d3993a917842979c51dc7c09fabd9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:24:23.011659 23655 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21d3993a917842979c51dc7c09fabd9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21d3993a917842979c51dc7c09fabd9b, State: Initialized, Role: FOLLOWER
I20260925 15:24:23.011809 23655 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21d3993a917842979c51dc7c09fabd9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21d3993a917842979c51dc7c09fabd9b" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38181 } }
I20260925 15:24:23.011914 23655 raft_consensus.cc:388] T 00000000000000000000000000000000 P 21d3993a917842979c51dc7c09fabd9b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:24:23.011950 23655 raft_consensus.cc:482] T 00000000000000000000000000000000 P 21d3993a917842979c51dc7c09fabd9b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:24:23.012023 23655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21d3993a917842979c51dc7c09fabd9b [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:23.012845 23655 raft_consensus.cc:504] T 00000000000000000000000000000000 P 21d3993a917842979c51dc7c09fabd9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21d3993a917842979c51dc7c09fabd9b" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38181 } }
I20260925 15:24:23.012985 23655 leader_election.cc:302] T 00000000000000000000000000000000 P 21d3993a917842979c51dc7c09fabd9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21d3993a917842979c51dc7c09fabd9b; no voters:
I20260925 15:24:23.013230 23655 leader_election.cc:288] T 00000000000000000000000000000000 P 21d3993a917842979c51dc7c09fabd9b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:24:23.013403 23660 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 21d3993a917842979c51dc7c09fabd9b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:24:23.013664 23660 raft_consensus.cc:686] T 00000000000000000000000000000000 P 21d3993a917842979c51dc7c09fabd9b [term 1 LEADER]: Becoming Leader. State: Replica: 21d3993a917842979c51dc7c09fabd9b, State: Running, Role: LEADER
I20260925 15:24:23.013818 23660 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21d3993a917842979c51dc7c09fabd9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21d3993a917842979c51dc7c09fabd9b" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38181 } }
I20260925 15:24:23.013914 23658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:23.005739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51510 (local address 127.3.211.254:38181)
0925 15:24:23.006023 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:23.006036 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:23.006076 (+ 40us) server_negotiation.cc:407] Connection header received
0925 15:24:23.006728 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:23.006736 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:23.006831 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:23.007182 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:23.007723 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:23.008856 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:23.010267 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:23.010997 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:23.011172 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:23.012588 (+ 1416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:23.012593 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:23.012594 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:24:23.012849 (+ 255us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:23.012898 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:24:23.012914 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:23.013028 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:23.013334 (+ 306us) server_negotiation.cc:299] Negotiation successful
0925 15:24:23.013466 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":132,"threads_started":1}
I20260925 15:24:23.013590 23655 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21d3993a917842979c51dc7c09fabd9b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:24:23.015368 23661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21d3993a917842979c51dc7c09fabd9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "21d3993a917842979c51dc7c09fabd9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21d3993a917842979c51dc7c09fabd9b" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38181 } } }
I20260925 15:24:23.015455 23662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21d3993a917842979c51dc7c09fabd9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21d3993a917842979c51dc7c09fabd9b. Latest consensus state: current_term: 1 leader_uuid: "21d3993a917842979c51dc7c09fabd9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21d3993a917842979c51dc7c09fabd9b" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 38181 } } }
I20260925 15:24:23.015501 23662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21d3993a917842979c51dc7c09fabd9b [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:23.015501 23661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21d3993a917842979c51dc7c09fabd9b [sys.catalog]: This master's current role is: LEADER
W20260925 15:24:23.016160 23675 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 21d3993a917842979c51dc7c09fabd9b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:24:23.016207 23675 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:24:23.016278 23676 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:24:23.016515 23676 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:24:23.018280 23676 catalog_manager.cc:1347] Generated new cluster ID: f294a9b86e044b7a8375728fc709b222
I20260925 15:24:23.018358 23676 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:24:23.050794 23676 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:24:23.051919 23676 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:24:23.059392 23676 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 21d3993a917842979c51dc7c09fabd9b: Generated new TSK 0
I20260925 15:24:23.059633 23676 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:24:23.062803 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.193:0
--local_ip_for_outbound_sockets=127.3.211.193
--webserver_interface=127.3.211.193
--webserver_port=0
--tserver_master_addrs=127.3.211.254:38181
--builtin_ntp_servers=127.3.211.212:37785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:24:23.159825 23679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:23.160101 23679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:23.160148 23679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:23.161999 23679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:23.162242 23679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:23.162330 23679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.193
I20260925 15:24:23.164662 23679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:37785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.211.193
--webserver_port=0
--tserver_master_addrs=127.3.211.254:38181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.193
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:24:23.164959 23679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:23.165295 23679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:23.171257 23685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:23.171257 23687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:23.171295 23684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:23.172761 23679 server_base.cc:1034] running on GCE node
I20260925 15:24:23.173208 23679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:23.173554 23679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:23.174748 23679 hybrid_clock.cc:648] HybridClock initialized: now 1790349863174716 us; error 55 us; skew 500 ppm
I20260925 15:24:23.176209 23679 webserver.cc:460] Webserver started at http://127.3.211.193:35041/ using document root <none> and password file <none>
I20260925 15:24:23.176554 23679 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:23.176642 23679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:23.176815 23679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:23.178838 23679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/ts-0/data/instance:
uuid: "b2af6e52fa78414eb584b53231ac9d51"
format_stamp: "Formatted at 2026-09-25 15:24:23 on dist-test-slave-4bk3"
server_key: "90c9946b2829bc854a70fbd0a6cce3fb"
server_key_iv: "ecdeaf45ca06fe9db275541745706c96"
server_key_version: "encryptionkey@0"
I20260925 15:24:23.179389 23679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/ts-0/wal/instance:
uuid: "b2af6e52fa78414eb584b53231ac9d51"
format_stamp: "Formatted at 2026-09-25 15:24:23 on dist-test-slave-4bk3"
server_key: "90c9946b2829bc854a70fbd0a6cce3fb"
server_key_iv: "ecdeaf45ca06fe9db275541745706c96"
server_key_version: "encryptionkey@0"
I20260925 15:24:23.181663 23679 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.005s sys 0.000s
I20260925 15:24:23.182514 23693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:23.182704 23679 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:24:23.182801 23679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/ts-0/wal
uuid: "b2af6e52fa78414eb584b53231ac9d51"
format_stamp: "Formatted at 2026-09-25 15:24:23 on dist-test-slave-4bk3"
server_key: "90c9946b2829bc854a70fbd0a6cce3fb"
server_key_iv: "ecdeaf45ca06fe9db275541745706c96"
server_key_version: "encryptionkey@0"
I20260925 15:24:23.182888 23679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:23.200781 23679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:23.201109 23679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:23.201225 23679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:23.201427 23679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:24:23.201810 23679 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:24:23.201850 23679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:23.201876 23679 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:24:23.201889 23679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:23.208674 23806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.193:46343 every 8 connection(s)
I20260925 15:24:23.208719 23679 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.193:46343
I20260925 15:24:23.209232 23679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:24:23.216599 23658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:23.210241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:34907 (local address 127.3.211.254:38181)
0925 15:24:23.210308 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:23.210311 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:23.210430 (+ 119us) server_negotiation.cc:407] Connection header received
0925 15:24:23.210969 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:23.210970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:23.210983 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:23.211070 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:23.211897 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:23.212537 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:23.214130 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:23.214505 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:23.214647 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:23.216137 (+ 1490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:23.216140 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:23.216140 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:24:23.216147 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:23.216169 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:24:23.216175 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:23.216245 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:23.216481 (+ 236us) server_negotiation.cc:299] Negotiation successful
0925 15:24:23.216529 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 15:24:23.216904 23809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:23.210075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38181 (local address 127.3.211.193:34907)
0925 15:24:23.210294 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:23.210338 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:23.210842 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:23.211186 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:23.211190 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:23.211200 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:24:23.211764 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:23.211769 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:23.212752 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:23.212754 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:23.213991 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:23.213997 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:23.214675 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:23.214677 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:23.214946 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:23.215912 (+ 966us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:24:23.215914 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:24:23.215967 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:24:23.216001 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:24:23.216303 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:24:23.216312 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:24:23.216315 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:24:23.216432 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 15:24:23.216578 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":115,"threads_started":1}
I20260925 15:24:23.217876 23807 heartbeater.cc:347] Connected to a master server at 127.3.211.254:38181
I20260925 15:24:23.218009 23807 heartbeater.cc:464] Registering TS with master...
I20260925 15:24:23.218247 23807 heartbeater.cc:511] Master 127.3.211.254:38181 requested a full tablet report, sending...
I20260925 15:24:23.218904 23619 ts_manager.cc:194] Registered new tserver with Master: b2af6e52fa78414eb584b53231ac9d51 (127.3.211.193:46343)
I20260925 15:24:23.219345 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 23679
I20260925 15:24:23.219462 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/ts-0/wal/instance
I20260925 15:24:23.219682 3919 external_mini_cluster.cc:1428] Setting key bae3be41020396af605ad1fa8ce6c9d1
I20260925 15:24:23.220067 23619 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.211.193:34907
I20260925 15:24:23.221400 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.194:0
--local_ip_for_outbound_sockets=127.3.211.194
--webserver_interface=127.3.211.194
--webserver_port=0
--tserver_master_addrs=127.3.211.254:38181
--builtin_ntp_servers=127.3.211.212:37785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:24:23.310468 23810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:23.311254 23810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:23.311401 23810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:23.313773 23810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:23.313856 23810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:23.313915 23810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.194
I20260925 15:24:23.315869 23810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:37785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.211.194
--webserver_port=0
--tserver_master_addrs=127.3.211.254:38181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.194
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:24:23.316226 23810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:23.316648 23810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:23.322233 23816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:23.322245 23815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:23.322245 23818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:23.322439 23810 server_base.cc:1034] running on GCE node
I20260925 15:24:23.322891 23810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:23.323216 23810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:23.324520 23810 hybrid_clock.cc:648] HybridClock initialized: now 1790349863324438 us; error 89 us; skew 500 ppm
I20260925 15:24:23.326557 23810 webserver.cc:460] Webserver started at http://127.3.211.194:39769/ using document root <none> and password file <none>
I20260925 15:24:23.326877 23810 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:23.326958 23810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:23.327167 23810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:23.329095 23810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/ts-1/data/instance:
uuid: "644810f3b4bb475ba59f7d8178b503b5"
format_stamp: "Formatted at 2026-09-25 15:24:23 on dist-test-slave-4bk3"
server_key: "d395ea26c0d71c7d1a39c25e857591b4"
server_key_iv: "5a34fb4e40d14902bb16184e4e5125c1"
server_key_version: "encryptionkey@0"
I20260925 15:24:23.329634 23810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/ts-1/wal/instance:
uuid: "644810f3b4bb475ba59f7d8178b503b5"
format_stamp: "Formatted at 2026-09-25 15:24:23 on dist-test-slave-4bk3"
server_key: "d395ea26c0d71c7d1a39c25e857591b4"
server_key_iv: "5a34fb4e40d14902bb16184e4e5125c1"
server_key_version: "encryptionkey@0"
I20260925 15:24:23.331755 23810 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:24:23.332760 23824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:23.334050 23810 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260925 15:24:23.334167 23810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/ts-1/wal
uuid: "644810f3b4bb475ba59f7d8178b503b5"
format_stamp: "Formatted at 2026-09-25 15:24:23 on dist-test-slave-4bk3"
server_key: "d395ea26c0d71c7d1a39c25e857591b4"
server_key_iv: "5a34fb4e40d14902bb16184e4e5125c1"
server_key_version: "encryptionkey@0"
I20260925 15:24:23.334261 23810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:23.347939 23810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:23.348286 23810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:23.348505 23810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:23.348753 23810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:24:23.349292 23810 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:24:23.349370 23810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:23.349416 23810 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:24:23.349443 23810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:23.355733 23810 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.194:43907
I20260925 15:24:23.355777 23937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.194:43907 every 8 connection(s)
I20260925 15:24:23.356163 23810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/ts-1/data/info.pb
I20260925 15:24:23.357237 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 23810
I20260925 15:24:23.357352 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/ts-1/wal/instance
I20260925 15:24:23.357555 3919 external_mini_cluster.cc:1428] Setting key f9bfc00ceafd36573013e874af5fbb9e
I20260925 15:24:23.358492 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.211.195:0
--local_ip_for_outbound_sockets=127.3.211.195
--webserver_interface=127.3.211.195
--webserver_port=0
--tserver_master_addrs=127.3.211.254:38181
--builtin_ntp_servers=127.3.211.212:37785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 15:24:23.362841 23658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:23.357032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.194:46391 (local address 127.3.211.254:38181)
0925 15:24:23.357109 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:23.357112 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:23.357334 (+ 222us) server_negotiation.cc:407] Connection header received
0925 15:24:23.357985 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:23.357986 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:23.357998 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:23.358044 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:23.358844 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:23.359355 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:23.360959 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:23.361274 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:23.361366 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:23.362411 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:23.362413 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:23.362414 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:24:23.362420 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:23.362440 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:24:23.362446 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:23.362515 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:23.362663 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 15:24:23.362703 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 15:24:23.363011 23940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:23.356987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38181 (local address 127.3.211.194:46391)
0925 15:24:23.357175 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:23.357217 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:23.357888 (+ 671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:23.358112 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:23.358116 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:23.358126 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:24:23.358734 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:23.358740 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:23.359469 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:23.359470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:23.360851 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:23.360856 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:23.361416 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:23.361417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:23.361563 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:23.362190 (+ 627us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:24:23.362191 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:24:23.362256 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:24:23.362294 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:24:23.362551 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:24:23.362561 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:24:23.362564 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:24:23.362654 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 15:24:23.362757 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":86,"threads_started":1}
I20260925 15:24:23.363679 23938 heartbeater.cc:347] Connected to a master server at 127.3.211.254:38181
I20260925 15:24:23.363804 23938 heartbeater.cc:464] Registering TS with master...
I20260925 15:24:23.364707 23938 heartbeater.cc:511] Master 127.3.211.254:38181 requested a full tablet report, sending...
I20260925 15:24:23.365268 23618 ts_manager.cc:194] Registered new tserver with Master: 644810f3b4bb475ba59f7d8178b503b5 (127.3.211.194:43907)
I20260925 15:24:23.365695 23618 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.211.194:46391
W20260925 15:24:23.449625 23941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:23.449914 23941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:23.449955 23941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:23.451222 23941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:23.451315 23941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:23.451365 23941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.211.195
I20260925 15:24:23.452943 23941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:37785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.211.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.211.195
--webserver_port=0
--tserver_master_addrs=127.3.211.254:38181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.211.195
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:24:23.453246 23941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:23.453584 23941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:23.458962 23949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:23.458962 23946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:23.459127 23947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:23.459653 23941 server_base.cc:1034] running on GCE node
I20260925 15:24:23.460093 23941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:23.460629 23941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:23.461849 23941 hybrid_clock.cc:648] HybridClock initialized: now 1790349863461821 us; error 49 us; skew 500 ppm
I20260925 15:24:23.463730 23941 webserver.cc:460] Webserver started at http://127.3.211.195:36737/ using document root <none> and password file <none>
I20260925 15:24:23.464042 23941 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:23.464126 23941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:23.464289 23941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:23.465497 23941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/ts-2/data/instance:
uuid: "40093fa4860348529d6dc1feccf00a4d"
format_stamp: "Formatted at 2026-09-25 15:24:23 on dist-test-slave-4bk3"
server_key: "4380379528df6cb829911c97bafa5a33"
server_key_iv: "3adc4ca21e266bd499531c0d98c4146a"
server_key_version: "encryptionkey@0"
I20260925 15:24:23.465917 23941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/ts-2/wal/instance:
uuid: "40093fa4860348529d6dc1feccf00a4d"
format_stamp: "Formatted at 2026-09-25 15:24:23 on dist-test-slave-4bk3"
server_key: "4380379528df6cb829911c97bafa5a33"
server_key_iv: "3adc4ca21e266bd499531c0d98c4146a"
server_key_version: "encryptionkey@0"
I20260925 15:24:23.467700 23941 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:24:23.468734 23955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:23.468964 23941 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:24:23.469076 23941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/ts-2/wal
uuid: "40093fa4860348529d6dc1feccf00a4d"
format_stamp: "Formatted at 2026-09-25 15:24:23 on dist-test-slave-4bk3"
server_key: "4380379528df6cb829911c97bafa5a33"
server_key_iv: "3adc4ca21e266bd499531c0d98c4146a"
server_key_version: "encryptionkey@0"
I20260925 15:24:23.469177 23941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:23.498481 23941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:23.498824 23941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:23.498978 23941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:23.499231 23941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:24:23.499591 23941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:24:23.499645 23941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:23.499681 23941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:24:23.499707 23941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:23.506309 23941 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.195:36849
I20260925 15:24:23.506417 24068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.195:36849 every 8 connection(s)
I20260925 15:24:23.506958 23941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/ts-2/data/info.pb
I20260925 15:24:23.514336 23658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:23.508037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:50349 (local address 127.3.211.254:38181)
0925 15:24:23.508141 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:23.508146 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:23.508655 (+ 509us) server_negotiation.cc:407] Connection header received
0925 15:24:23.509298 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:23.509300 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:23.509313 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:23.509379 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:23.510495 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:23.511044 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:23.512279 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:23.512590 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:23.512661 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:23.513758 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:23.513760 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:23.513760 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:24:23.513765 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:23.513782 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:24:23.513786 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:23.513837 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:23.514162 (+ 325us) server_negotiation.cc:299] Negotiation successful
0925 15:24:23.514223 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 15:24:23.514559 24071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:23.508025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:38181 (local address 127.3.211.195:50349)
0925 15:24:23.508456 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:23.508505 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:23.509140 (+ 635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:23.509575 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:23.509580 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:23.509592 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:24:23.510311 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:23.510319 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:23.511276 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:23.511277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:23.512151 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:23.512154 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:23.512719 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:23.512720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:23.512851 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:23.513571 (+ 720us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:24:23.513571 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:24:23.513613 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:24:23.513649 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:24:23.513876 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:24:23.513883 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:24:23.513886 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:24:23.513967 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 15:24:23.514247 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":141,"threads_started":1}
I20260925 15:24:23.515201 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 23941
I20260925 15:24:23.515321 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/ts-2/wal/instance
I20260925 15:24:23.515326 24069 heartbeater.cc:347] Connected to a master server at 127.3.211.254:38181
I20260925 15:24:23.515484 3919 external_mini_cluster.cc:1428] Setting key 69aa1dbf02f5469203bb36bd90d07019
I20260925 15:24:23.515508 24069 heartbeater.cc:464] Registering TS with master...
I20260925 15:24:23.515923 24069 heartbeater.cc:511] Master 127.3.211.254:38181 requested a full tablet report, sending...
I20260925 15:24:23.516558 23619 ts_manager.cc:194] Registered new tserver with Master: 40093fa4860348529d6dc1feccf00a4d (127.3.211.195:36849)
I20260925 15:24:23.517167 23619 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.211.195:50349
I20260925 15:24:23.528779 3919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:24:23.533684 23658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:23.530381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51522 (local address 127.3.211.254:38181)
0925 15:24:23.530486 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:23.530489 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:23.530586 (+ 97us) server_negotiation.cc:407] Connection header received
0925 15:24:23.530694 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:23.530695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:23.530707 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:23.530794 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:23.530969 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:23.531506 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:23.532267 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:23.532584 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:23.532673 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:23.533405 (+ 732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:23.533406 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:23.533406 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:24:23.533412 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:23.533427 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:24:23.533431 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:23.533485 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:23.533552 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 15:24:23.533581 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 15:24:23.536077 23619 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51522:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:24:23.539651 23619 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:24:23.549504 24080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:23.543987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.195:36849 (local address 127.0.0.1:48322)
0925 15:24:23.544426 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:23.544436 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:23.544484 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:23.544905 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:23.544909 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:23.544912 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:24:23.545002 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:23.545005 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:23.546384 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:23.546386 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:23.547926 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:23.547950 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:23.549128 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:23.549128 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:23.549224 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:23.549240 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 15:24:23.549308 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 15:24:23.549384 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":76,"threads_started":1}
I20260925 15:24:23.549610 24078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:23.543983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:46343 (local address 127.0.0.1:59824)
0925 15:24:23.544416 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:23.544429 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:23.544489 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:23.545000 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:23.545001 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:23.545002 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:24:23.545054 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:23.545057 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:23.546624 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:23.546626 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:23.548008 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:23.548010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:23.549017 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:23.549019 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:23.549456 (+ 437us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:23.549461 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:24:23.549495 (+ 34us) client_negotiation.cc:240] Negotiation successful
0925 15:24:23.549537 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"mutex_wait_us":64,"thread_start_us":95,"threads_started":1}
I20260925 15:24:23.549906 24079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:23.544136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40892 (local address 127.3.211.194:43907)
0925 15:24:23.545197 (+ 1061us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:23.545202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:23.545212 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:24:23.545248 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:23.545250 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:23.545265 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:23.545346 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:24:23.545499 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:23.546569 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:23.548081 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:23.549140 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:23.549237 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:23.549332 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:23.549409 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 15:24:23.549717 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1022,"thread_start_us":87,"threads_started":1}
I20260925 15:24:23.550001 24081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:23.544258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59824 (local address 127.3.211.193:46343)
0925 15:24:23.544786 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:23.544790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:23.544799 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:24:23.544835 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:23.544837 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:23.544851 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:23.544940 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:24:23.545145 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:23.546472 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:23.548098 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:23.548911 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:23.549689 (+ 778us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:23.549803 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:23.549831 (+ 28us) server_negotiation.cc:299] Negotiation successful
0925 15:24:23.549881 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":136,"threads_started":1}
I20260925 15:24:23.549752 24077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:23.543983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.194:43907 (local address 127.0.0.1:40892)
0925 15:24:23.544119 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:23.544250 (+ 131us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:23.544392 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:23.545352 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:23.545353 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:23.545356 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:24:23.545407 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:23.545408 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:23.546719 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:23.546719 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:23.547969 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:23.547972 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:23.549237 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:23.549238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:23.549307 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:23.549312 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:24:23.549384 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 15:24:23.549605 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":2,"thread_start_us":315,"threads_started":1}
I20260925 15:24:23.550812 24082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:23.544489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48322 (local address 127.3.211.195:36849)
0925 15:24:23.544652 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:23.544658 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:23.544668 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:24:23.544711 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:23.544714 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:23.544733 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:23.544814 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:24:23.545089 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:23.546279 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:23.548064 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:23.548902 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:23.548985 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:23.549094 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:23.550668 (+ 1574us) server_negotiation.cc:299] Negotiation successful
0925 15:24:23.550725 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":96,"threads_started":1}
I20260925 15:24:23.551137 23741 tablet_service.cc:1467] Processing CreateTablet for tablet b35e99bee9bc42e48480e3278e940fc1 (DEFAULT_TABLE table=test-table [id=5d692a25aabc4424a0bf09f8a34c5049]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:24:23.551139 23872 tablet_service.cc:1467] Processing CreateTablet for tablet b35e99bee9bc42e48480e3278e940fc1 (DEFAULT_TABLE table=test-table [id=5d692a25aabc4424a0bf09f8a34c5049]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:24:23.551534 23741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b35e99bee9bc42e48480e3278e940fc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:23.551537 23872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b35e99bee9bc42e48480e3278e940fc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:23.551638 24003 tablet_service.cc:1467] Processing CreateTablet for tablet b35e99bee9bc42e48480e3278e940fc1 (DEFAULT_TABLE table=test-table [id=5d692a25aabc4424a0bf09f8a34c5049]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:24:23.551944 24003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b35e99bee9bc42e48480e3278e940fc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:23.555421 24084 tablet_bootstrap.cc:492] T b35e99bee9bc42e48480e3278e940fc1 P 40093fa4860348529d6dc1feccf00a4d: Bootstrap starting.
I20260925 15:24:23.556406 24084 tablet_bootstrap.cc:654] T b35e99bee9bc42e48480e3278e940fc1 P 40093fa4860348529d6dc1feccf00a4d: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:23.556823 24084 log.cc:824] T b35e99bee9bc42e48480e3278e940fc1 P 40093fa4860348529d6dc1feccf00a4d: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:23.556789 24085 tablet_bootstrap.cc:492] T b35e99bee9bc42e48480e3278e940fc1 P 644810f3b4bb475ba59f7d8178b503b5: Bootstrap starting.
I20260925 15:24:23.557291 24083 tablet_bootstrap.cc:492] T b35e99bee9bc42e48480e3278e940fc1 P b2af6e52fa78414eb584b53231ac9d51: Bootstrap starting.
I20260925 15:24:23.557704 24085 tablet_bootstrap.cc:654] T b35e99bee9bc42e48480e3278e940fc1 P 644810f3b4bb475ba59f7d8178b503b5: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:23.557986 24085 log.cc:824] T b35e99bee9bc42e48480e3278e940fc1 P 644810f3b4bb475ba59f7d8178b503b5: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:23.558362 24083 tablet_bootstrap.cc:654] T b35e99bee9bc42e48480e3278e940fc1 P b2af6e52fa78414eb584b53231ac9d51: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:23.558830 24083 log.cc:824] T b35e99bee9bc42e48480e3278e940fc1 P b2af6e52fa78414eb584b53231ac9d51: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:23.559016 24085 tablet_bootstrap.cc:492] T b35e99bee9bc42e48480e3278e940fc1 P 644810f3b4bb475ba59f7d8178b503b5: No bootstrap required, opened a new log
I20260925 15:24:23.559106 24085 ts_tablet_manager.cc:1397] T b35e99bee9bc42e48480e3278e940fc1 P 644810f3b4bb475ba59f7d8178b503b5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:24:23.559465 24084 tablet_bootstrap.cc:492] T b35e99bee9bc42e48480e3278e940fc1 P 40093fa4860348529d6dc1feccf00a4d: No bootstrap required, opened a new log
I20260925 15:24:23.559543 24084 ts_tablet_manager.cc:1397] T b35e99bee9bc42e48480e3278e940fc1 P 40093fa4860348529d6dc1feccf00a4d: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260925 15:24:23.562028 24085 raft_consensus.cc:348] T b35e99bee9bc42e48480e3278e940fc1 P 644810f3b4bb475ba59f7d8178b503b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2af6e52fa78414eb584b53231ac9d51" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 46343 } } peers { permanent_uuid: "644810f3b4bb475ba59f7d8178b503b5" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 43907 } } peers { permanent_uuid: "40093fa4860348529d6dc1feccf00a4d" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 36849 } }
I20260925 15:24:23.562258 24085 raft_consensus.cc:374] T b35e99bee9bc42e48480e3278e940fc1 P 644810f3b4bb475ba59f7d8178b503b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:24:23.562304 24085 raft_consensus.cc:729] T b35e99bee9bc42e48480e3278e940fc1 P 644810f3b4bb475ba59f7d8178b503b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 644810f3b4bb475ba59f7d8178b503b5, State: Initialized, Role: FOLLOWER
I20260925 15:24:23.562278 24084 raft_consensus.cc:348] T b35e99bee9bc42e48480e3278e940fc1 P 40093fa4860348529d6dc1feccf00a4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2af6e52fa78414eb584b53231ac9d51" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 46343 } } peers { permanent_uuid: "644810f3b4bb475ba59f7d8178b503b5" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 43907 } } peers { permanent_uuid: "40093fa4860348529d6dc1feccf00a4d" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 36849 } }
I20260925 15:24:23.562427 24084 raft_consensus.cc:374] T b35e99bee9bc42e48480e3278e940fc1 P 40093fa4860348529d6dc1feccf00a4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:24:23.562484 24084 raft_consensus.cc:729] T b35e99bee9bc42e48480e3278e940fc1 P 40093fa4860348529d6dc1feccf00a4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40093fa4860348529d6dc1feccf00a4d, State: Initialized, Role: FOLLOWER
I20260925 15:24:23.562496 24085 consensus_queue.cc:260] T b35e99bee9bc42e48480e3278e940fc1 P 644810f3b4bb475ba59f7d8178b503b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2af6e52fa78414eb584b53231ac9d51" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 46343 } } peers { permanent_uuid: "644810f3b4bb475ba59f7d8178b503b5" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 43907 } } peers { permanent_uuid: "40093fa4860348529d6dc1feccf00a4d" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 36849 } }
I20260925 15:24:23.562604 24084 consensus_queue.cc:260] T b35e99bee9bc42e48480e3278e940fc1 P 40093fa4860348529d6dc1feccf00a4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2af6e52fa78414eb584b53231ac9d51" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 46343 } } peers { permanent_uuid: "644810f3b4bb475ba59f7d8178b503b5" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 43907 } } peers { permanent_uuid: "40093fa4860348529d6dc1feccf00a4d" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 36849 } }
I20260925 15:24:23.562829 24085 ts_tablet_manager.cc:1428] T b35e99bee9bc42e48480e3278e940fc1 P 644810f3b4bb475ba59f7d8178b503b5: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260925 15:24:23.562844 24084 ts_tablet_manager.cc:1428] T b35e99bee9bc42e48480e3278e940fc1 P 40093fa4860348529d6dc1feccf00a4d: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260925 15:24:23.562983 23938 heartbeater.cc:503] Master 127.3.211.254:38181 was elected leader, sending a full tablet report...
I20260925 15:24:23.563084 24069 heartbeater.cc:503] Master 127.3.211.254:38181 was elected leader, sending a full tablet report...
I20260925 15:24:23.564697 24083 tablet_bootstrap.cc:492] T b35e99bee9bc42e48480e3278e940fc1 P b2af6e52fa78414eb584b53231ac9d51: No bootstrap required, opened a new log
I20260925 15:24:23.564854 24083 ts_tablet_manager.cc:1397] T b35e99bee9bc42e48480e3278e940fc1 P b2af6e52fa78414eb584b53231ac9d51: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20260925 15:24:23.567612 24083 raft_consensus.cc:348] T b35e99bee9bc42e48480e3278e940fc1 P b2af6e52fa78414eb584b53231ac9d51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2af6e52fa78414eb584b53231ac9d51" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 46343 } } peers { permanent_uuid: "644810f3b4bb475ba59f7d8178b503b5" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 43907 } } peers { permanent_uuid: "40093fa4860348529d6dc1feccf00a4d" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 36849 } }
I20260925 15:24:23.567894 24083 raft_consensus.cc:374] T b35e99bee9bc42e48480e3278e940fc1 P b2af6e52fa78414eb584b53231ac9d51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:24:23.567922 24083 raft_consensus.cc:729] T b35e99bee9bc42e48480e3278e940fc1 P b2af6e52fa78414eb584b53231ac9d51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2af6e52fa78414eb584b53231ac9d51, State: Initialized, Role: FOLLOWER
I20260925 15:24:23.568130 24083 consensus_queue.cc:260] T b35e99bee9bc42e48480e3278e940fc1 P b2af6e52fa78414eb584b53231ac9d51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2af6e52fa78414eb584b53231ac9d51" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 46343 } } peers { permanent_uuid: "644810f3b4bb475ba59f7d8178b503b5" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 43907 } } peers { permanent_uuid: "40093fa4860348529d6dc1feccf00a4d" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 36849 } }
I20260925 15:24:23.568614 24083 ts_tablet_manager.cc:1428] T b35e99bee9bc42e48480e3278e940fc1 P b2af6e52fa78414eb584b53231ac9d51: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20260925 15:24:23.568706 23807 heartbeater.cc:503] Master 127.3.211.254:38181 was elected leader, sending a full tablet report...
W20260925 15:24:23.606829 23939 tablet.cc:2369] T b35e99bee9bc42e48480e3278e940fc1 P 644810f3b4bb475ba59f7d8178b503b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:24:23.710232 23808 tablet.cc:2369] T b35e99bee9bc42e48480e3278e940fc1 P b2af6e52fa78414eb584b53231ac9d51: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:24:23.757191 24091 raft_consensus.cc:482] T b35e99bee9bc42e48480e3278e940fc1 P b2af6e52fa78414eb584b53231ac9d51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:24:23.757335 24091 raft_consensus.cc:504] T b35e99bee9bc42e48480e3278e940fc1 P b2af6e52fa78414eb584b53231ac9d51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2af6e52fa78414eb584b53231ac9d51" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 46343 } } peers { permanent_uuid: "644810f3b4bb475ba59f7d8178b503b5" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 43907 } } peers { permanent_uuid: "40093fa4860348529d6dc1feccf00a4d" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 36849 } }
I20260925 15:24:23.757731 24091 leader_election.cc:288] T b35e99bee9bc42e48480e3278e940fc1 P b2af6e52fa78414eb584b53231ac9d51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 644810f3b4bb475ba59f7d8178b503b5 (127.3.211.194:43907), 40093fa4860348529d6dc1feccf00a4d (127.3.211.195:36849)
W20260925 15:24:23.757795 24070 tablet.cc:2369] T b35e99bee9bc42e48480e3278e940fc1 P 40093fa4860348529d6dc1feccf00a4d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:24:23.760947 24082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:23.758087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:60503 (local address 127.3.211.195:36849)
0925 15:24:23.758131 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:23.758134 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:23.758168 (+ 34us) server_negotiation.cc:407] Connection header received
0925 15:24:23.758287 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:23.758288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:23.758298 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:23.758374 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:24:23.758547 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:23.759030 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:23.760116 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:23.760563 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:23.760650 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:23.760659 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:23.760821 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 15:24:23.760866 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 15:24:23.760986 24079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:23.757992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.193:41811 (local address 127.3.211.194:43907)
0925 15:24:23.758043 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:23.758045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:23.758148 (+ 103us) server_negotiation.cc:407] Connection header received
0925 15:24:23.758294 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:23.758296 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:23.758308 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:23.758400 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:24:23.758562 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:23.759025 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:23.760128 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:23.760618 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:23.760702 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:23.760711 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:23.760872 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 15:24:23.760911 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 15:24:23.761005 24093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:23.757938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.195:36849 (local address 127.3.211.193:60503)
0925 15:24:23.758101 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:23.758114 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:23.758187 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:23.758377 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:23.758378 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:23.758380 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:24:23.758464 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:23.758466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:23.759119 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:23.759120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:23.760009 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:23.760011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:23.760650 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:23.760651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:23.760732 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:23.760736 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:24:23.760813 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 15:24:23.760860 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":51,"threads_started":1}
I20260925 15:24:23.761204 24092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:23.757905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.194:43907 (local address 127.3.211.193:41811)
0925 15:24:23.758072 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:23.758085 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:23.758195 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:23.758403 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:23.758404 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:23.758406 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:24:23.758486 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:23.758488 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:23.759122 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:23.759123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:23.760029 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:23.760031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:23.760711 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:23.760713 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:23.760787 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:23.760791 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:24:23.760862 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 15:24:23.760904 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":50,"thread_start_us":67,"threads_started":1}
I20260925 15:24:23.761391 24090 raft_consensus.cc:482] T b35e99bee9bc42e48480e3278e940fc1 P 40093fa4860348529d6dc1feccf00a4d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:24:23.761377 24023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b35e99bee9bc42e48480e3278e940fc1" candidate_uuid: "b2af6e52fa78414eb584b53231ac9d51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40093fa4860348529d6dc1feccf00a4d" is_pre_election: true
I20260925 15:24:23.761497 24090 raft_consensus.cc:504] T b35e99bee9bc42e48480e3278e940fc1 P 40093fa4860348529d6dc1feccf00a4d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2af6e52fa78414eb584b53231ac9d51" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 46343 } } peers { permanent_uuid: "644810f3b4bb475ba59f7d8178b503b5" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 43907 } } peers { permanent_uuid: "40093fa4860348529d6dc1feccf00a4d" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 36849 } }
I20260925 15:24:23.761610 24023 raft_consensus.cc:2454] T b35e99bee9bc42e48480e3278e940fc1 P 40093fa4860348529d6dc1feccf00a4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2af6e52fa78414eb584b53231ac9d51 in term 0.
I20260925 15:24:23.761605 23892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b35e99bee9bc42e48480e3278e940fc1" candidate_uuid: "b2af6e52fa78414eb584b53231ac9d51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "644810f3b4bb475ba59f7d8178b503b5" is_pre_election: true
I20260925 15:24:23.761778 23892 raft_consensus.cc:2454] T b35e99bee9bc42e48480e3278e940fc1 P 644810f3b4bb475ba59f7d8178b503b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2af6e52fa78414eb584b53231ac9d51 in term 0.
I20260925 15:24:23.761987 24090 leader_election.cc:288] T b35e99bee9bc42e48480e3278e940fc1 P 40093fa4860348529d6dc1feccf00a4d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b2af6e52fa78414eb584b53231ac9d51 (127.3.211.193:46343), 644810f3b4bb475ba59f7d8178b503b5 (127.3.211.194:43907)
I20260925 15:24:23.762153 23695 leader_election.cc:302] T b35e99bee9bc42e48480e3278e940fc1 P b2af6e52fa78414eb584b53231ac9d51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 644810f3b4bb475ba59f7d8178b503b5, b2af6e52fa78414eb584b53231ac9d51; no voters:
I20260925 15:24:23.762336 24091 raft_consensus.cc:2781] T b35e99bee9bc42e48480e3278e940fc1 P b2af6e52fa78414eb584b53231ac9d51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:24:23.762399 24091 raft_consensus.cc:482] T b35e99bee9bc42e48480e3278e940fc1 P b2af6e52fa78414eb584b53231ac9d51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:24:23.762430 24091 raft_consensus.cc:3037] T b35e99bee9bc42e48480e3278e940fc1 P b2af6e52fa78414eb584b53231ac9d51 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:23.763461 24091 raft_consensus.cc:504] T b35e99bee9bc42e48480e3278e940fc1 P b2af6e52fa78414eb584b53231ac9d51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2af6e52fa78414eb584b53231ac9d51" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 46343 } } peers { permanent_uuid: "644810f3b4bb475ba59f7d8178b503b5" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 43907 } } peers { permanent_uuid: "40093fa4860348529d6dc1feccf00a4d" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 36849 } }
I20260925 15:24:23.763619 24091 leader_election.cc:288] T b35e99bee9bc42e48480e3278e940fc1 P b2af6e52fa78414eb584b53231ac9d51 [CANDIDATE]: Term 1 election: Requested vote from peers 644810f3b4bb475ba59f7d8178b503b5 (127.3.211.194:43907), 40093fa4860348529d6dc1feccf00a4d (127.3.211.195:36849)
I20260925 15:24:23.763845 23892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b35e99bee9bc42e48480e3278e940fc1" candidate_uuid: "b2af6e52fa78414eb584b53231ac9d51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "644810f3b4bb475ba59f7d8178b503b5"
I20260925 15:24:23.763870 24023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b35e99bee9bc42e48480e3278e940fc1" candidate_uuid: "b2af6e52fa78414eb584b53231ac9d51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40093fa4860348529d6dc1feccf00a4d"
I20260925 15:24:23.763931 23892 raft_consensus.cc:3037] T b35e99bee9bc42e48480e3278e940fc1 P 644810f3b4bb475ba59f7d8178b503b5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:23.763936 24023 raft_consensus.cc:3037] T b35e99bee9bc42e48480e3278e940fc1 P 40093fa4860348529d6dc1feccf00a4d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:23.764890 23892 raft_consensus.cc:2454] T b35e99bee9bc42e48480e3278e940fc1 P 644810f3b4bb475ba59f7d8178b503b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2af6e52fa78414eb584b53231ac9d51 in term 1.
I20260925 15:24:23.764883 24023 raft_consensus.cc:2454] T b35e99bee9bc42e48480e3278e940fc1 P 40093fa4860348529d6dc1feccf00a4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2af6e52fa78414eb584b53231ac9d51 in term 1.
I20260925 15:24:23.765050 23695 leader_election.cc:302] T b35e99bee9bc42e48480e3278e940fc1 P b2af6e52fa78414eb584b53231ac9d51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 644810f3b4bb475ba59f7d8178b503b5, b2af6e52fa78414eb584b53231ac9d51; no voters:
I20260925 15:24:23.765141 24091 raft_consensus.cc:2781] T b35e99bee9bc42e48480e3278e940fc1 P b2af6e52fa78414eb584b53231ac9d51 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:24:23.765309 24091 raft_consensus.cc:686] T b35e99bee9bc42e48480e3278e940fc1 P b2af6e52fa78414eb584b53231ac9d51 [term 1 LEADER]: Becoming Leader. State: Replica: b2af6e52fa78414eb584b53231ac9d51, State: Running, Role: LEADER
I20260925 15:24:23.765407 24091 consensus_queue.cc:237] T b35e99bee9bc42e48480e3278e940fc1 P b2af6e52fa78414eb584b53231ac9d51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2af6e52fa78414eb584b53231ac9d51" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 46343 } } peers { permanent_uuid: "644810f3b4bb475ba59f7d8178b503b5" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 43907 } } peers { permanent_uuid: "40093fa4860348529d6dc1feccf00a4d" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 36849 } }
I20260925 15:24:23.766039 24071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:23.762202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:46343 (local address 127.3.211.195:42049)
0925 15:24:23.762286 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:23.762298 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:23.762345 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:23.762648 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:23.762649 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:23.762651 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:24:23.762741 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:23.762743 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:23.763630 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:23.763637 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:23.764863 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:23.764866 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:23.765782 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:23.765783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:23.765855 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:23.765859 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:24:23.765917 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 15:24:23.765970 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 15:24:23.766052 24081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:23.762448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:42049 (local address 127.3.211.193:46343)
0925 15:24:23.762488 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:23.762490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:23.762497 (+ 7us) server_negotiation.cc:407] Connection header received
0925 15:24:23.762519 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:23.762520 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:23.762529 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:23.762600 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:24:23.762840 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:23.763505 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:23.765107 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:23.765629 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:23.765693 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:23.765703 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:23.765953 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 15:24:23.765993 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 15:24:23.766386 23761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b35e99bee9bc42e48480e3278e940fc1" candidate_uuid: "40093fa4860348529d6dc1feccf00a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2af6e52fa78414eb584b53231ac9d51" is_pre_election: true
I20260925 15:24:23.766412 23617 catalog_manager.cc:5495] T b35e99bee9bc42e48480e3278e940fc1 P b2af6e52fa78414eb584b53231ac9d51 reported cstate change: term changed from 0 to 1, leader changed from <none> to b2af6e52fa78414eb584b53231ac9d51 (127.3.211.193). New cstate: current_term: 1 leader_uuid: "b2af6e52fa78414eb584b53231ac9d51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2af6e52fa78414eb584b53231ac9d51" member_type: VOTER last_known_addr { host: "127.3.211.193" port: 46343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "644810f3b4bb475ba59f7d8178b503b5" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 43907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40093fa4860348529d6dc1feccf00a4d" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 36849 } health_report { overall_health: UNKNOWN } } }
I20260925 15:24:23.767130 24079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:23.762395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.211.195:54239 (local address 127.3.211.194:43907)
0925 15:24:23.762509 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:23.762511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:23.762627 (+ 116us) server_negotiation.cc:407] Connection header received
0925 15:24:23.762735 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:23.762736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:23.762745 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:23.762803 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:24:23.763772 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:23.764519 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:23.766280 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:23.766773 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:23.766822 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:23.766829 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:23.767033 (+ 204us) server_negotiation.cc:299] Negotiation successful
0925 15:24:23.767069 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260925 15:24:23.767146 24094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:23.762228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.194:43907 (local address 127.3.211.195:54239)
0925 15:24:23.762571 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:23.762582 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:23.762650 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:23.763618 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:23.763619 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:23.763621 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:24:23.763692 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:23.763694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:23.765068 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:23.765069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:23.766179 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:23.766182 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:23.766898 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:23.766899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:23.766951 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:23.766955 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:24:23.767027 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 15:24:23.767067 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":115,"threads_started":1}
I20260925 15:24:23.767366 23892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b35e99bee9bc42e48480e3278e940fc1" candidate_uuid: "40093fa4860348529d6dc1feccf00a4d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "644810f3b4bb475ba59f7d8178b503b5" is_pre_election: true
I20260925 15:24:23.767503 23892 raft_consensus.cc:2379] T b35e99bee9bc42e48480e3278e940fc1 P 644810f3b4bb475ba59f7d8178b503b5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 40093fa4860348529d6dc1feccf00a4d in current term 1: Already voted for candidate b2af6e52fa78414eb584b53231ac9d51 in this term.
I20260925 15:24:23.767740 23957 leader_election.cc:302] T b35e99bee9bc42e48480e3278e940fc1 P 40093fa4860348529d6dc1feccf00a4d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 40093fa4860348529d6dc1feccf00a4d; no voters: 644810f3b4bb475ba59f7d8178b503b5, b2af6e52fa78414eb584b53231ac9d51
I20260925 15:24:23.767889 24090 raft_consensus.cc:2726] T b35e99bee9bc42e48480e3278e940fc1 P 40093fa4860348529d6dc1feccf00a4d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 15:24:23.783026 24081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:23.780394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59826 (local address 127.3.211.193:46343)
0925 15:24:23.780486 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:23.780488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:23.780499 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:24:23.780536 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:23.780537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:23.780556 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:23.780620 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:24:23.780739 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:23.781208 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:23.781987 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:23.782177 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:23.782243 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:23.782489 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:24:23.782601 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:24:23.782683 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:23.782917 (+ 234us) server_negotiation.cc:299] Negotiation successful
0925 15:24:23.782945 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 15:24:23.785229 23892 raft_consensus.cc:1261] T b35e99bee9bc42e48480e3278e940fc1 P 644810f3b4bb475ba59f7d8178b503b5 [term 1 FOLLOWER]: Refusing update from remote peer b2af6e52fa78414eb584b53231ac9d51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:24:23.785277 24023 raft_consensus.cc:1261] T b35e99bee9bc42e48480e3278e940fc1 P 40093fa4860348529d6dc1feccf00a4d [term 1 FOLLOWER]: Refusing update from remote peer b2af6e52fa78414eb584b53231ac9d51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:24:23.785662 24095 consensus_queue.cc:1035] T b35e99bee9bc42e48480e3278e940fc1 P b2af6e52fa78414eb584b53231ac9d51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "644810f3b4bb475ba59f7d8178b503b5" member_type: VOTER last_known_addr { host: "127.3.211.194" port: 43907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:24:23.785775 24091 consensus_queue.cc:1035] T b35e99bee9bc42e48480e3278e940fc1 P b2af6e52fa78414eb584b53231ac9d51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40093fa4860348529d6dc1feccf00a4d" member_type: VOTER last_known_addr { host: "127.3.211.195" port: 36849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:24:23.788293 24098 mvcc.cc:205] Tried to move back new op lower bound from 7333273042061209600 to 7333273041983655936. Current Snapshot: MvccSnapshot[applied={T|T < 7333273042061209600}]
I20260925 15:24:23.788894 24100 mvcc.cc:205] Tried to move back new op lower bound from 7333273042061209600 to 7333273041983655936. Current Snapshot: MvccSnapshot[applied={T|T < 7333273042061209600}]
I20260925 15:24:23.789428 24099 mvcc.cc:205] Tried to move back new op lower bound from 7333273042061209600 to 7333273041983655936. Current Snapshot: MvccSnapshot[applied={T|T < 7333273042061209600 or (T in {7333273042061209600})}]
I20260925 15:24:23.790920 23618 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:24:23.791011 23618 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:24:23.791847 23618 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 21d3993a917842979c51dc7c09fabd9b: Sending DeleteTablet for 3 replicas of tablet b35e99bee9bc42e48480e3278e940fc1
I20260925 15:24:23.792238 23872 tablet_service.cc:1514] Processing DeleteTablet for tablet b35e99bee9bc42e48480e3278e940fc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:24:23 UTC) from {username='slave'} at 127.0.0.1:40892
I20260925 15:24:23.792404 24003 tablet_service.cc:1514] Processing DeleteTablet for tablet b35e99bee9bc42e48480e3278e940fc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:24:23 UTC) from {username='slave'} at 127.0.0.1:48322
I20260925 15:24:23.792550 23741 tablet_service.cc:1514] Processing DeleteTablet for tablet b35e99bee9bc42e48480e3278e940fc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:24:23 UTC) from {username='slave'} at 127.0.0.1:59824
I20260925 15:24:23.792599 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 23679
I20260925 15:24:23.793363 24107 tablet_replica.cc:330] T b35e99bee9bc42e48480e3278e940fc1 P 644810f3b4bb475ba59f7d8178b503b5: stopping tablet replica
I20260925 15:24:23.793500 24107 raft_consensus.cc:2229] T b35e99bee9bc42e48480e3278e940fc1 P 644810f3b4bb475ba59f7d8178b503b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:24:23.793725 24107 raft_consensus.cc:2258] T b35e99bee9bc42e48480e3278e940fc1 P 644810f3b4bb475ba59f7d8178b503b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:24:23.794312 24107 ts_tablet_manager.cc:1905] T b35e99bee9bc42e48480e3278e940fc1 P 644810f3b4bb475ba59f7d8178b503b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:24:23.794727 24108 tablet_replica.cc:330] T b35e99bee9bc42e48480e3278e940fc1 P 40093fa4860348529d6dc1feccf00a4d: stopping tablet replica
I20260925 15:24:23.794963 24108 raft_consensus.cc:2229] T b35e99bee9bc42e48480e3278e940fc1 P 40093fa4860348529d6dc1feccf00a4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:24:23.795203 24108 raft_consensus.cc:2258] T b35e99bee9bc42e48480e3278e940fc1 P 40093fa4860348529d6dc1feccf00a4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:24:23.795868 24108 ts_tablet_manager.cc:1905] T b35e99bee9bc42e48480e3278e940fc1 P 40093fa4860348529d6dc1feccf00a4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:24:23.796708 24107 ts_tablet_manager.cc:1918] T b35e99bee9bc42e48480e3278e940fc1 P 644810f3b4bb475ba59f7d8178b503b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:24:23.796805 24107 log.cc:1196] T b35e99bee9bc42e48480e3278e940fc1 P 644810f3b4bb475ba59f7d8178b503b5: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/ts-1/wal/wals/b35e99bee9bc42e48480e3278e940fc1
I20260925 15:24:23.797026 24107 ts_tablet_manager.cc:1939] T b35e99bee9bc42e48480e3278e940fc1 P 644810f3b4bb475ba59f7d8178b503b5: Deleting consensus metadata
I20260925 15:24:23.797729 23604 catalog_manager.cc:4841] TS 644810f3b4bb475ba59f7d8178b503b5 (127.3.211.194:43907): tablet b35e99bee9bc42e48480e3278e940fc1 (table test-table [id=5d692a25aabc4424a0bf09f8a34c5049]) successfully deleted
I20260925 15:24:23.798753 24108 ts_tablet_manager.cc:1918] T b35e99bee9bc42e48480e3278e940fc1 P 40093fa4860348529d6dc1feccf00a4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:24:23.798926 24108 log.cc:1196] T b35e99bee9bc42e48480e3278e940fc1 P 40093fa4860348529d6dc1feccf00a4d: Deleting WAL directory at /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790349610595785-3919-0/minicluster-data/ts-2/wal/wals/b35e99bee9bc42e48480e3278e940fc1
W20260925 15:24:23.799012 23606 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.211.193:46343 (error 108)
I20260925 15:24:23.799292 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 23810
I20260925 15:24:23.799335 24108 ts_tablet_manager.cc:1939] T b35e99bee9bc42e48480e3278e940fc1 P 40093fa4860348529d6dc1feccf00a4d: Deleting consensus metadata
I20260925 15:24:23.800038 24080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:23.799425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.193:46343 (local address 127.0.0.1:59832)
0925 15:24:23.799942 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:23.799991 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.193:46343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494}
I20260925 15:24:23.800414 23603 catalog_manager.cc:4841] TS 40093fa4860348529d6dc1feccf00a4d (127.3.211.195:36849): tablet b35e99bee9bc42e48480e3278e940fc1 (table test-table [id=5d692a25aabc4424a0bf09f8a34c5049]) successfully deleted
W20260925 15:24:23.801149 23606 catalog_manager.cc:4584] TS b2af6e52fa78414eb584b53231ac9d51 (127.3.211.193:46343): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b35e99bee9bc42e48480e3278e940fc1: Network error: Client connection negotiation failed: client connection to 127.3.211.193:46343: connect: Connection refused (error 111)
I20260925 15:24:23.804526 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 23941
I20260925 15:24:23.809088 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 23588
2026-09-25T15:24:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (979 ms)
[----------] 6 tests from EncryptionPolicyTest (9475 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349610595785-3919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:24:23 dist-test-slave-4bk3 krb5kdc[24112](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:24:23 dist-test-slave-4bk3 krb5kdc[24112](info): set up 2 sockets
Sep 25 15:24:23 dist-test-slave-4bk3 krb5kdc[24112](info): commencing operation
krb5kdc: starting...
W20260925 15:24:24.872929 3919 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Sep 25 15:24:24 dist-test-slave-4bk3 krb5kdc[24112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:24:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:24:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349610595785-3919-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349610595785-3919-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:24:25.012053 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35707
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349610595785-3919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35707
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:24:25.100903 24125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:25.101141 24125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:25.101167 24125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:25.102351 24125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:24:25.102401 24125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:25.102415 24125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:25.102428 24125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:24:25.102469 24125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:24:25.104014 24125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349610595785-3919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35707
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349610595785-3919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:24:25.104238 24125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:25.104548 24125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:25.109772 24130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:25.109929 24125 server_base.cc:1034] running on GCE node
W20260925 15:24:25.110002 24133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:25.109766 24131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:25.110358 24125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:25.110711 24125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:25.111932 24125 hybrid_clock.cc:648] HybridClock initialized: now 1790349865111920 us; error 98 us; skew 500 ppm
Sep 25 15:24:25 dist-test-slave-4bk3 krb5kdc[24112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349865, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:24:25.121570 24125 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:24:25.122020 24125 webserver.cc:460] Webserver started at http://127.0.0.1:43543/ using document root <none> and password file <none>
I20260925 15:24:25.122241 24125 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:25.122284 24125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:25.122400 24125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:25.123468 24125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349610595785-3919-0/minicluster-data/master-0/data/instance:
uuid: "d0fcde356e1f4bef82e365188da18dd2"
format_stamp: "Formatted at 2026-09-25 15:24:25 on dist-test-slave-4bk3"
server_key: "20f44c828e41e9103a4f8d8ddc408237"
server_key_iv: "f9c3e05bdf324c5da7023f2809fe3b32"
server_key_version: "encryptionkey@0"
I20260925 15:24:25.123798 24125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349610595785-3919-0/minicluster-data/master-0/wal/instance:
uuid: "d0fcde356e1f4bef82e365188da18dd2"
format_stamp: "Formatted at 2026-09-25 15:24:25 on dist-test-slave-4bk3"
server_key: "20f44c828e41e9103a4f8d8ddc408237"
server_key_iv: "f9c3e05bdf324c5da7023f2809fe3b32"
server_key_version: "encryptionkey@0"
I20260925 15:24:25.125244 24125 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:24:25.126130 24140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:25.126324 24125 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:24:25.126416 24125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349610595785-3919-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349610595785-3919-0/minicluster-data/master-0/wal
uuid: "d0fcde356e1f4bef82e365188da18dd2"
format_stamp: "Formatted at 2026-09-25 15:24:25 on dist-test-slave-4bk3"
server_key: "20f44c828e41e9103a4f8d8ddc408237"
server_key_iv: "f9c3e05bdf324c5da7023f2809fe3b32"
server_key_version: "encryptionkey@0"
I20260925 15:24:25.126480 24125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349610595785-3919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349610595785-3919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349610595785-3919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:25.135375 24125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:25.137878 24125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:25.138008 24125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:25.142058 24125 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35707
I20260925 15:24:25.142175 24192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35707 every 8 connection(s)
I20260925 15:24:25.142508 24125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
I20260925 15:24:25.143677 24193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:25.146327 24193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0fcde356e1f4bef82e365188da18dd2: Bootstrap starting.
I20260925 15:24:25.147029 24193 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0fcde356e1f4bef82e365188da18dd2: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:25.147346 24193 log.cc:824] T 00000000000000000000000000000000 P d0fcde356e1f4bef82e365188da18dd2: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:25.148030 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 24125
I20260925 15:24:25.148118 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790349610595785-3919-0/minicluster-data/master-0/wal/instance
I20260925 15:24:25.148226 24193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0fcde356e1f4bef82e365188da18dd2: No bootstrap required, opened a new log
I20260925 15:24:25.148350 3919 external_mini_cluster.cc:1428] Setting key 0ade66a8a46bc33a1065a7a7f66aa81d
I20260925 15:24:25.150884 24193 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0fcde356e1f4bef82e365188da18dd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0fcde356e1f4bef82e365188da18dd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35707 } }
I20260925 15:24:25.151091 24193 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0fcde356e1f4bef82e365188da18dd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:24:25.151134 24193 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0fcde356e1f4bef82e365188da18dd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0fcde356e1f4bef82e365188da18dd2, State: Initialized, Role: FOLLOWER
I20260925 15:24:25.151309 24193 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0fcde356e1f4bef82e365188da18dd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0fcde356e1f4bef82e365188da18dd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35707 } }
I20260925 15:24:25.151422 24193 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0fcde356e1f4bef82e365188da18dd2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:24:25.151464 24193 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0fcde356e1f4bef82e365188da18dd2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:24:25.151511 24193 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0fcde356e1f4bef82e365188da18dd2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:25.152415 24193 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0fcde356e1f4bef82e365188da18dd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0fcde356e1f4bef82e365188da18dd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35707 } }
I20260925 15:24:25.152545 24193 leader_election.cc:302] T 00000000000000000000000000000000 P d0fcde356e1f4bef82e365188da18dd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0fcde356e1f4bef82e365188da18dd2; no voters:
I20260925 15:24:25.152717 24193 leader_election.cc:288] T 00000000000000000000000000000000 P d0fcde356e1f4bef82e365188da18dd2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:24:25.152792 24198 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0fcde356e1f4bef82e365188da18dd2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:24:25.152959 24198 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0fcde356e1f4bef82e365188da18dd2 [term 1 LEADER]: Becoming Leader. State: Replica: d0fcde356e1f4bef82e365188da18dd2, State: Running, Role: LEADER
I20260925 15:24:25.153005 24193 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0fcde356e1f4bef82e365188da18dd2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:24:25.153046 24198 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0fcde356e1f4bef82e365188da18dd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0fcde356e1f4bef82e365188da18dd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35707 } }
I20260925 15:24:25.153401 24198 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0fcde356e1f4bef82e365188da18dd2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0fcde356e1f4bef82e365188da18dd2. Latest consensus state: current_term: 1 leader_uuid: "d0fcde356e1f4bef82e365188da18dd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0fcde356e1f4bef82e365188da18dd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35707 } } }
I20260925 15:24:25.153477 24198 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0fcde356e1f4bef82e365188da18dd2 [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:25.153694 24204 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:24:25.154263 24204 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:24:25.154686 24199 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0fcde356e1f4bef82e365188da18dd2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0fcde356e1f4bef82e365188da18dd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0fcde356e1f4bef82e365188da18dd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35707 } } }
I20260925 15:24:25.154834 24199 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0fcde356e1f4bef82e365188da18dd2 [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:25.156323 24204 catalog_manager.cc:1347] Generated new cluster ID: 7a32243074624be8be24fd6ad7106b3b
I20260925 15:24:25.156430 24204 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 15:24:25 dist-test-slave-4bk3 krb5kdc[24112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 15:24:25.163640 24197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:25.149024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41943 (local address 127.0.0.1:35707)
0925 15:24:25.149251 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:25.149261 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:25.149299 (+ 38us) server_negotiation.cc:407] Connection header received
0925 15:24:25.150058 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:25.150067 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:25.150139 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:25.150316 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:25.151461 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:25.153245 (+ 1784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:25.154454 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:25.155113 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:25.155227 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:25.158622 (+ 3395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:25.158626 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:25.158627 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:25.158642 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:25.160825 (+ 2183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:25.161338 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:25.161341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:25.161342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:25.161461 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:25.161707 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:25.161708 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:25.161708 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:25.162063 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:25.162962 (+ 899us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:25.163145 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 15:24:25.163324 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":114,"threads_started":1}
I20260925 15:24:25.165226 3919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:24:25.169734 24204 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:24:25.170480 24204 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:24:25.173393 24204 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0fcde356e1f4bef82e365188da18dd2: Generated new TSK 0
I20260925 15:24:25.173623 24204 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:24:25.176697 24197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:25.167233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34825 (local address 127.0.0.1:35707)
0925 15:24:25.167290 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:25.167292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:25.167572 (+ 280us) server_negotiation.cc:407] Connection header received
0925 15:24:25.167622 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:25.167623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:25.167638 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:25.167884 (+ 246us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:25.168519 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:25.169335 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:25.170341 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:25.170698 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:25.170779 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:25.172390 (+ 1611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:25.172391 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:25.172392 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:25.172400 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:25.174548 (+ 2148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:25.175067 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:25.175068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:25.175068 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:25.175125 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:25.175424 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:25.175426 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:25.175426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:25.175722 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:25.175883 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:25.176472 (+ 589us) server_negotiation.cc:299] Negotiation successful
0925 15:24:25.176568 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 15:24:25.178972 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 24125
W20260925 15:24:25.184394 24110 connection.cc:535] client connection to 127.0.0.1:35707 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35707): Connection reset by peer (error 104)
2026-09-25T15:24:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1369 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349610595785-3919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:24:25 dist-test-slave-4bk3 krb5kdc[24224](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:24:25 dist-test-slave-4bk3 krb5kdc[24224](info): set up 2 sockets
Sep 25 15:24:25 dist-test-slave-4bk3 krb5kdc[24224](info): commencing operation
krb5kdc: starting...
W20260925 15:24:26.244529 3919 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Sep 25 15:24:26 dist-test-slave-4bk3 krb5kdc[24224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349866, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:24:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:24:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349610595785-3919-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349610595785-3919-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:24:26.377820 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42805
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349610595785-3919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:24:26.469744 24237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:26.470001 24237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:26.470031 24237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:26.471539 24237 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:24:26.471614 24237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:26.471642 24237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:26.471655 24237 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:24:26.471668 24237 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:24:26.473368 24237 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349610595785-3919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42805
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349610595785-3919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:24:26.473619 24237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:26.473912 24237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:26.479261 24242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:26.479290 24243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:26.479303 24245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:26.479808 24237 server_base.cc:1034] running on GCE node
I20260925 15:24:26.480178 24237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:26.480667 24237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:26.481925 24237 hybrid_clock.cc:648] HybridClock initialized: now 1790349866481895 us; error 38 us; skew 500 ppm
Sep 25 15:24:26 dist-test-slave-4bk3 krb5kdc[24224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349866, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:24:26.485754 24237 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:24:26.486524 24237 webserver.cc:460] Webserver started at http://127.0.0.1:39441/ using document root <none> and password file <none>
I20260925 15:24:26.486848 24237 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:26.486933 24237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:26.487129 24237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:26.488749 24237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349610595785-3919-0/minicluster-data/master-0/data/instance:
uuid: "a998c1b5b1cc4f509c89a1fcb27f05ec"
format_stamp: "Formatted at 2026-09-25 15:24:26 on dist-test-slave-4bk3"
server_key: "914081cddd09789ed3de187e4a0009cb"
server_key_iv: "ad79dee5516e3e06d3d56e1d3de7cd3b"
server_key_version: "encryptionkey@0"
I20260925 15:24:26.489271 24237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349610595785-3919-0/minicluster-data/master-0/wal/instance:
uuid: "a998c1b5b1cc4f509c89a1fcb27f05ec"
format_stamp: "Formatted at 2026-09-25 15:24:26 on dist-test-slave-4bk3"
server_key: "914081cddd09789ed3de187e4a0009cb"
server_key_iv: "ad79dee5516e3e06d3d56e1d3de7cd3b"
server_key_version: "encryptionkey@0"
I20260925 15:24:26.491297 24237 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:24:26.492504 24252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:26.492843 24237 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:24:26.493021 24237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349610595785-3919-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349610595785-3919-0/minicluster-data/master-0/wal
uuid: "a998c1b5b1cc4f509c89a1fcb27f05ec"
format_stamp: "Formatted at 2026-09-25 15:24:26 on dist-test-slave-4bk3"
server_key: "914081cddd09789ed3de187e4a0009cb"
server_key_iv: "ad79dee5516e3e06d3d56e1d3de7cd3b"
server_key_version: "encryptionkey@0"
I20260925 15:24:26.493217 24237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349610595785-3919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349610595785-3919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349610595785-3919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:26.514111 24237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:26.516691 24237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:26.516871 24237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:26.521306 24237 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42805
I20260925 15:24:26.521373 24304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42805 every 8 connection(s)
I20260925 15:24:26.521766 24237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
I20260925 15:24:26.522842 24305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:26.524767 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 24237
I20260925 15:24:26.524909 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790349610595785-3919-0/minicluster-data/master-0/wal/instance
I20260925 15:24:26.525132 3919 external_mini_cluster.cc:1428] Setting key bb6aabe7f72352b4f9f43254602a23e1
I20260925 15:24:26.525905 24305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a998c1b5b1cc4f509c89a1fcb27f05ec: Bootstrap starting.
I20260925 15:24:26.526582 24305 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a998c1b5b1cc4f509c89a1fcb27f05ec: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:26.526962 24305 log.cc:824] T 00000000000000000000000000000000 P a998c1b5b1cc4f509c89a1fcb27f05ec: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:26.527830 24305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a998c1b5b1cc4f509c89a1fcb27f05ec: No bootstrap required, opened a new log
I20260925 15:24:26.530290 24305 raft_consensus.cc:348] T 00000000000000000000000000000000 P a998c1b5b1cc4f509c89a1fcb27f05ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a998c1b5b1cc4f509c89a1fcb27f05ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42805 } }
I20260925 15:24:26.530505 24305 raft_consensus.cc:374] T 00000000000000000000000000000000 P a998c1b5b1cc4f509c89a1fcb27f05ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:24:26.530543 24305 raft_consensus.cc:729] T 00000000000000000000000000000000 P a998c1b5b1cc4f509c89a1fcb27f05ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a998c1b5b1cc4f509c89a1fcb27f05ec, State: Initialized, Role: FOLLOWER
I20260925 15:24:26.530727 24305 consensus_queue.cc:260] T 00000000000000000000000000000000 P a998c1b5b1cc4f509c89a1fcb27f05ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a998c1b5b1cc4f509c89a1fcb27f05ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42805 } }
I20260925 15:24:26.530831 24305 raft_consensus.cc:388] T 00000000000000000000000000000000 P a998c1b5b1cc4f509c89a1fcb27f05ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:24:26.530879 24305 raft_consensus.cc:482] T 00000000000000000000000000000000 P a998c1b5b1cc4f509c89a1fcb27f05ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:24:26.530929 24305 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a998c1b5b1cc4f509c89a1fcb27f05ec [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:26.531567 24308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:26.525749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42353 (local address 127.0.0.1:42805)
0925 15:24:26.525985 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:26.525995 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:26.526032 (+ 37us) server_negotiation.cc:407] Connection header received
0925 15:24:26.526489 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:26.526496 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:26.526543 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:26.526719 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:26.527069 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:26.528223 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:26.529386 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:26.529819 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:26.529917 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:26.530790 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:26.530794 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:26.530794 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:24:26.531055 (+ 261us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:26.531089 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:24:26.531100 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:26.531174 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:26.531306 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 15:24:26.531352 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":103,"threads_started":1}
I20260925 15:24:26.531715 24305 raft_consensus.cc:504] T 00000000000000000000000000000000 P a998c1b5b1cc4f509c89a1fcb27f05ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a998c1b5b1cc4f509c89a1fcb27f05ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42805 } }
I20260925 15:24:26.531823 24305 leader_election.cc:302] T 00000000000000000000000000000000 P a998c1b5b1cc4f509c89a1fcb27f05ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a998c1b5b1cc4f509c89a1fcb27f05ec; no voters:
I20260925 15:24:26.532002 24305 leader_election.cc:288] T 00000000000000000000000000000000 P a998c1b5b1cc4f509c89a1fcb27f05ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:24:26.532059 24310 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a998c1b5b1cc4f509c89a1fcb27f05ec [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:24:26.532241 24310 raft_consensus.cc:686] T 00000000000000000000000000000000 P a998c1b5b1cc4f509c89a1fcb27f05ec [term 1 LEADER]: Becoming Leader. State: Replica: a998c1b5b1cc4f509c89a1fcb27f05ec, State: Running, Role: LEADER
I20260925 15:24:26.532363 24305 sys_catalog.cc:564] T 00000000000000000000000000000000 P a998c1b5b1cc4f509c89a1fcb27f05ec [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:24:26.532387 24310 consensus_queue.cc:237] T 00000000000000000000000000000000 P a998c1b5b1cc4f509c89a1fcb27f05ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a998c1b5b1cc4f509c89a1fcb27f05ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42805 } }
I20260925 15:24:26.532508 3919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:24:26.533634 24311 sys_catalog.cc:455] T 00000000000000000000000000000000 P a998c1b5b1cc4f509c89a1fcb27f05ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a998c1b5b1cc4f509c89a1fcb27f05ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a998c1b5b1cc4f509c89a1fcb27f05ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42805 } } }
I20260925 15:24:26.533730 24311 sys_catalog.cc:458] T 00000000000000000000000000000000 P a998c1b5b1cc4f509c89a1fcb27f05ec [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:26.533965 24312 sys_catalog.cc:455] T 00000000000000000000000000000000 P a998c1b5b1cc4f509c89a1fcb27f05ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader a998c1b5b1cc4f509c89a1fcb27f05ec. Latest consensus state: current_term: 1 leader_uuid: "a998c1b5b1cc4f509c89a1fcb27f05ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a998c1b5b1cc4f509c89a1fcb27f05ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42805 } } }
I20260925 15:24:26.534148 24312 sys_catalog.cc:458] T 00000000000000000000000000000000 P a998c1b5b1cc4f509c89a1fcb27f05ec [sys.catalog]: This master's current role is: LEADER
W20260925 15:24:26.534425 24328 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a998c1b5b1cc4f509c89a1fcb27f05ec: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:24:26.534511 24328 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:24:26.534582 24330 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:24:26.534808 24330 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:24:26.536612 24330 catalog_manager.cc:1347] Generated new cluster ID: f82a3d9467d540d3b6adb1bbb4caaf6e
I20260925 15:24:26.536666 24330 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:24:26.538043 24308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:26.534413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39899 (local address 127.0.0.1:42805)
0925 15:24:26.534449 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:26.534451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:26.534461 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:24:26.534554 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:26.534555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:26.534565 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:26.534595 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:26.534802 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:26.535530 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:26.536570 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:26.536879 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:26.536938 (+ 59us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:26.537757 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:26.537758 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:26.537758 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:24:26.537764 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:26.537781 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:24:26.537785 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:26.537843 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:26.537956 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 15:24:26.537971 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 15:24:26.550199 24330 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:24:26.550797 24330 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:24:26.561985 24330 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a998c1b5b1cc4f509c89a1fcb27f05ec: Generated new TSK 0
I20260925 15:24:26.562304 24330 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:24:26.570144 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 24237
W20260925 15:24:26.574656 24222 connection.cc:535] client connection to 127.0.0.1:42805 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42805): Connection reset by peer (error 104)
2026-09-25T15:24:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1390 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349610595785-3919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:24:26 dist-test-slave-4bk3 krb5kdc[24336](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:24:26 dist-test-slave-4bk3 krb5kdc[24336](info): set up 2 sockets
Sep 25 15:24:26 dist-test-slave-4bk3 krb5kdc[24336](info): commencing operation
krb5kdc: starting...
W20260925 15:24:27.657178 3919 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Sep 25 15:24:27 dist-test-slave-4bk3 krb5kdc[24336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:24:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:24:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349610595785-3919-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349610595785-3919-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:24:27.796970 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42695
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349610595785-3919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42695
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:24:27.883627 24349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:27.883910 24349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:27.883947 24349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:27.885257 24349 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:24:27.885349 24349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:27.885373 24349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:27.885396 24349 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:24:27.885432 24349 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:24:27.887071 24349 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349610595785-3919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42695
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349610595785-3919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:24:27.887403 24349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:27.887668 24349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:27.893309 24355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:27.893322 24357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:27.893422 24349 server_base.cc:1034] running on GCE node
W20260925 15:24:27.893309 24354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:27.893927 24349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:27.894266 24349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:27.895469 24349 hybrid_clock.cc:648] HybridClock initialized: now 1790349867895452 us; error 33 us; skew 500 ppm
Sep 25 15:24:27 dist-test-slave-4bk3 krb5kdc[24336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349867, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:24:27.898527 24349 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:24:27.899075 24349 webserver.cc:460] Webserver started at http://127.0.0.1:34525/ using document root <none> and password file <none>
I20260925 15:24:27.899366 24349 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:27.899420 24349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:27.899570 24349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:27.900882 24349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349610595785-3919-0/minicluster-data/master-0/data/instance:
uuid: "bb58761d698a42cebcdf946db7393242"
format_stamp: "Formatted at 2026-09-25 15:24:27 on dist-test-slave-4bk3"
server_key: "d72537adaa18c2c26a1344fab2b33fba"
server_key_iv: "b8106c5fa2d709d0ec7d5e48fbd7d7bf"
server_key_version: "encryptionkey@0"
I20260925 15:24:27.901319 24349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349610595785-3919-0/minicluster-data/master-0/wal/instance:
uuid: "bb58761d698a42cebcdf946db7393242"
format_stamp: "Formatted at 2026-09-25 15:24:27 on dist-test-slave-4bk3"
server_key: "d72537adaa18c2c26a1344fab2b33fba"
server_key_iv: "b8106c5fa2d709d0ec7d5e48fbd7d7bf"
server_key_version: "encryptionkey@0"
I20260925 15:24:27.902942 24349 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:24:27.903926 24364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:27.904521 24349 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:24:27.904642 24349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349610595785-3919-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349610595785-3919-0/minicluster-data/master-0/wal
uuid: "bb58761d698a42cebcdf946db7393242"
format_stamp: "Formatted at 2026-09-25 15:24:27 on dist-test-slave-4bk3"
server_key: "d72537adaa18c2c26a1344fab2b33fba"
server_key_iv: "b8106c5fa2d709d0ec7d5e48fbd7d7bf"
server_key_version: "encryptionkey@0"
I20260925 15:24:27.904753 24349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349610595785-3919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349610595785-3919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349610595785-3919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:27.908519 24349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:27.911595 24349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:27.911842 24349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:27.917116 24349 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42695
I20260925 15:24:27.917148 24416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42695 every 8 connection(s)
I20260925 15:24:27.917549 24349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
I20260925 15:24:27.918649 24417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:27.921289 24417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb58761d698a42cebcdf946db7393242: Bootstrap starting.
I20260925 15:24:27.921931 24417 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb58761d698a42cebcdf946db7393242: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:27.922190 24417 log.cc:824] T 00000000000000000000000000000000 P bb58761d698a42cebcdf946db7393242: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:27.922940 24417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb58761d698a42cebcdf946db7393242: No bootstrap required, opened a new log
I20260925 15:24:27.923763 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 24349
I20260925 15:24:27.923889 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790349610595785-3919-0/minicluster-data/master-0/wal/instance
I20260925 15:24:27.924117 3919 external_mini_cluster.cc:1428] Setting key fd0f1d878032e8e840396ed098991590
I20260925 15:24:27.925851 24417 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb58761d698a42cebcdf946db7393242 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb58761d698a42cebcdf946db7393242" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42695 } }
I20260925 15:24:27.926064 24417 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb58761d698a42cebcdf946db7393242 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:24:27.926115 24417 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb58761d698a42cebcdf946db7393242 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb58761d698a42cebcdf946db7393242, State: Initialized, Role: FOLLOWER
I20260925 15:24:27.926255 24417 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb58761d698a42cebcdf946db7393242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb58761d698a42cebcdf946db7393242" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42695 } }
I20260925 15:24:27.926373 24417 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb58761d698a42cebcdf946db7393242 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:24:27.926409 24417 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb58761d698a42cebcdf946db7393242 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:24:27.926460 24417 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb58761d698a42cebcdf946db7393242 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:27.927279 24421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:27.924930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45353 (local address 127.0.0.1:42695)
0925 15:24:27.925204 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:27.925213 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:27.925244 (+ 31us) server_negotiation.cc:407] Connection header received
0925 15:24:27.925832 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:27.925841 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:27.925874 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:27.926072 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:27.926785 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:27.926788 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:27.926789 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:24:27.926793 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:27.926847 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:24:27.926859 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:27.926940 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:27.927006 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 15:24:27.927053 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":158,"threads_started":1}
I20260925 15:24:27.927575 24417 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb58761d698a42cebcdf946db7393242 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb58761d698a42cebcdf946db7393242" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42695 } }
I20260925 15:24:27.927693 24417 leader_election.cc:302] T 00000000000000000000000000000000 P bb58761d698a42cebcdf946db7393242 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb58761d698a42cebcdf946db7393242; no voters:
I20260925 15:24:27.927891 24417 leader_election.cc:288] T 00000000000000000000000000000000 P bb58761d698a42cebcdf946db7393242 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:24:27.927917 24422 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb58761d698a42cebcdf946db7393242 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:24:27.928036 24422 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb58761d698a42cebcdf946db7393242 [term 1 LEADER]: Becoming Leader. State: Replica: bb58761d698a42cebcdf946db7393242, State: Running, Role: LEADER
I20260925 15:24:27.928117 24422 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb58761d698a42cebcdf946db7393242 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb58761d698a42cebcdf946db7393242" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42695 } }
I20260925 15:24:27.928176 24417 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb58761d698a42cebcdf946db7393242 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:24:27.928232 3919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:24:27.928489 24424 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb58761d698a42cebcdf946db7393242 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb58761d698a42cebcdf946db7393242. Latest consensus state: current_term: 1 leader_uuid: "bb58761d698a42cebcdf946db7393242" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb58761d698a42cebcdf946db7393242" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42695 } } }
I20260925 15:24:27.928577 24424 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb58761d698a42cebcdf946db7393242 [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:27.928493 24423 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb58761d698a42cebcdf946db7393242 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb58761d698a42cebcdf946db7393242" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb58761d698a42cebcdf946db7393242" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42695 } } }
I20260925 15:24:27.928720 24423 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb58761d698a42cebcdf946db7393242 [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:27.928889 24427 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:24:27.929489 24427 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:24:27.931519 24427 catalog_manager.cc:1347] Generated new cluster ID: b1002607e5f04fc5ac21b98cedf57c3a
I20260925 15:24:27.931609 24427 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:24:27.931730 24421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:27.929930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52501 (local address 127.0.0.1:42695)
0925 15:24:27.929965 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:27.929967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:27.930479 (+ 512us) server_negotiation.cc:407] Connection header received
0925 15:24:27.930546 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:27.930547 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:27.930553 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:27.930626 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:27.931338 (+ 712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:27.931339 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:27.931339 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:24:27.931344 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:27.931360 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:24:27.931364 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:27.931395 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:27.931502 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 15:24:27.931518 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 15:24:27.948997 24427 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:24:27.949242 24427 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:24:27.956492 24427 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb58761d698a42cebcdf946db7393242: Generated new TSK 0
I20260925 15:24:27.956740 24427 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:24:27.965257 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 24349
2026-09-25T15:24:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1395 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349610595785-3919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:24:28 dist-test-slave-4bk3 krb5kdc[24448](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:24:28 dist-test-slave-4bk3 krb5kdc[24448](info): set up 2 sockets
Sep 25 15:24:28 dist-test-slave-4bk3 krb5kdc[24448](info): commencing operation
krb5kdc: starting...
W20260925 15:24:29.044662 3919 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Sep 25 15:24:29 dist-test-slave-4bk3 krb5kdc[24448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:24:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:24:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349610595785-3919-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349610595785-3919-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:24:29.181003 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43087
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349610595785-3919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43087
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:24:29.273131 24461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:29.273433 24461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:29.273473 24461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:29.274739 24461 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:24:29.274837 24461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:29.274861 24461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:29.274883 24461 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:24:29.274912 24461 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:24:29.278040 24461 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349610595785-3919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43087
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349610595785-3919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:24:29.278532 24461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:29.278863 24461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:29.285247 24466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:29.285413 24467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:29.285245 24469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:29.285715 24461 server_base.cc:1034] running on GCE node
I20260925 15:24:29.286020 24461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:29.286482 24461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:29.287818 24461 hybrid_clock.cc:648] HybridClock initialized: now 1790349869287794 us; error 59 us; skew 500 ppm
Sep 25 15:24:29 dist-test-slave-4bk3 krb5kdc[24448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349869, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:24:29.290673 24461 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:24:29.291204 24461 webserver.cc:460] Webserver started at http://127.0.0.1:45467/ using document root <none> and password file <none>
I20260925 15:24:29.291553 24461 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:29.291651 24461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:29.291819 24461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:29.293167 24461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349610595785-3919-0/minicluster-data/master-0/data/instance:
uuid: "c0358cd638f94768ad2f249972ed3e13"
format_stamp: "Formatted at 2026-09-25 15:24:29 on dist-test-slave-4bk3"
server_key: "a047116342b19c1e9f28af1deedda21e"
server_key_iv: "90e5345161353051e39b0e124e44e4b1"
server_key_version: "encryptionkey@0"
I20260925 15:24:29.293563 24461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349610595785-3919-0/minicluster-data/master-0/wal/instance:
uuid: "c0358cd638f94768ad2f249972ed3e13"
format_stamp: "Formatted at 2026-09-25 15:24:29 on dist-test-slave-4bk3"
server_key: "a047116342b19c1e9f28af1deedda21e"
server_key_iv: "90e5345161353051e39b0e124e44e4b1"
server_key_version: "encryptionkey@0"
I20260925 15:24:29.295095 24461 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:24:29.296096 24476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:29.296398 24461 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:24:29.296761 24461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349610595785-3919-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349610595785-3919-0/minicluster-data/master-0/wal
uuid: "c0358cd638f94768ad2f249972ed3e13"
format_stamp: "Formatted at 2026-09-25 15:24:29 on dist-test-slave-4bk3"
server_key: "a047116342b19c1e9f28af1deedda21e"
server_key_iv: "90e5345161353051e39b0e124e44e4b1"
server_key_version: "encryptionkey@0"
I20260925 15:24:29.296945 24461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349610595785-3919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349610595785-3919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349610595785-3919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:29.324819 24461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:29.327291 24461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:29.327535 24461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:29.333289 24461 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43087
I20260925 15:24:29.333334 24528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43087 every 8 connection(s)
I20260925 15:24:29.333851 24461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
I20260925 15:24:29.334918 24529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:29.337556 24529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0358cd638f94768ad2f249972ed3e13: Bootstrap starting.
I20260925 15:24:29.337657 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 24461
I20260925 15:24:29.337766 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790349610595785-3919-0/minicluster-data/master-0/wal/instance
I20260925 15:24:29.337949 3919 external_mini_cluster.cc:1428] Setting key 8a6d3b49689bb634b5028537c4f78834
I20260925 15:24:29.338451 24529 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0358cd638f94768ad2f249972ed3e13: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:29.338815 24529 log.cc:824] T 00000000000000000000000000000000 P c0358cd638f94768ad2f249972ed3e13: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:29.339941 24529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0358cd638f94768ad2f249972ed3e13: No bootstrap required, opened a new log
I20260925 15:24:29.342468 24529 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0358cd638f94768ad2f249972ed3e13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0358cd638f94768ad2f249972ed3e13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43087 } }
I20260925 15:24:29.342675 24529 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0358cd638f94768ad2f249972ed3e13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:24:29.342715 24529 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0358cd638f94768ad2f249972ed3e13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0358cd638f94768ad2f249972ed3e13, State: Initialized, Role: FOLLOWER
I20260925 15:24:29.342898 24529 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0358cd638f94768ad2f249972ed3e13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0358cd638f94768ad2f249972ed3e13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43087 } }
I20260925 15:24:29.342991 24529 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0358cd638f94768ad2f249972ed3e13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:24:29.343024 24529 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0358cd638f94768ad2f249972ed3e13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:24:29.343063 24529 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0358cd638f94768ad2f249972ed3e13 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:29.343760 24529 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0358cd638f94768ad2f249972ed3e13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0358cd638f94768ad2f249972ed3e13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43087 } }
I20260925 15:24:29.343864 24529 leader_election.cc:302] T 00000000000000000000000000000000 P c0358cd638f94768ad2f249972ed3e13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0358cd638f94768ad2f249972ed3e13; no voters:
I20260925 15:24:29.344070 24529 leader_election.cc:288] T 00000000000000000000000000000000 P c0358cd638f94768ad2f249972ed3e13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:24:29.344132 24534 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0358cd638f94768ad2f249972ed3e13 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:24:29.344396 24534 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0358cd638f94768ad2f249972ed3e13 [term 1 LEADER]: Becoming Leader. State: Replica: c0358cd638f94768ad2f249972ed3e13, State: Running, Role: LEADER
I20260925 15:24:29.344466 24529 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0358cd638f94768ad2f249972ed3e13 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:24:29.344503 24534 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0358cd638f94768ad2f249972ed3e13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0358cd638f94768ad2f249972ed3e13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43087 } }
I20260925 15:24:29.344848 24534 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0358cd638f94768ad2f249972ed3e13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0358cd638f94768ad2f249972ed3e13. Latest consensus state: current_term: 1 leader_uuid: "c0358cd638f94768ad2f249972ed3e13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0358cd638f94768ad2f249972ed3e13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43087 } } }
I20260925 15:24:29.344913 24535 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0358cd638f94768ad2f249972ed3e13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0358cd638f94768ad2f249972ed3e13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0358cd638f94768ad2f249972ed3e13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43087 } } }
I20260925 15:24:29.344931 24534 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0358cd638f94768ad2f249972ed3e13 [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:29.345157 24535 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0358cd638f94768ad2f249972ed3e13 [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:29.345331 24541 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:24:29.346102 24541 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 15:24:29 dist-test-slave-4bk3 krb5kdc[24448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 15:24:29.348150 24541 catalog_manager.cc:1347] Generated new cluster ID: 27bc78de3994482b8ae3d1b800c18811
I20260925 15:24:29.348210 24541 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:24:29.351871 24532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:29.338838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45175 (local address 127.0.0.1:43087)
0925 15:24:29.339123 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:29.339142 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:29.339208 (+ 66us) server_negotiation.cc:407] Connection header received
0925 15:24:29.340204 (+ 996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:29.340215 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:29.340281 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:29.340499 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:29.341341 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:29.342272 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:29.343354 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:29.344013 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:29.344167 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:29.347787 (+ 3620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:29.347792 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:29.347793 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:29.347808 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:29.350004 (+ 2196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:29.350457 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:29.350462 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:29.350462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:29.350581 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:29.350790 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:29.350791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:29.350792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:29.351195 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:29.351377 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:29.351463 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 15:24:29.351613 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":149,"threads_started":1}
I20260925 15:24:29.353494 3919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:24:29.359099 24541 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:24:29.359786 24541 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:24:29.363551 24532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:29.355026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59653 (local address 127.0.0.1:43087)
0925 15:24:29.355066 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:29.355068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:29.355187 (+ 119us) server_negotiation.cc:407] Connection header received
0925 15:24:29.355251 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:29.355252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:29.355262 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:29.355330 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:29.356020 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:29.356829 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:29.357955 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:29.358268 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:29.358343 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:29.359976 (+ 1633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:29.359977 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:29.359977 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:29.359986 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:29.362098 (+ 2112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:29.362571 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:29.362572 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:29.362572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:29.362633 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:29.362947 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:29.362948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:29.362949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:29.363075 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:29.363177 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:29.363334 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 15:24:29.363447 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 15:24:29.363874 24541 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0358cd638f94768ad2f249972ed3e13: Generated new TSK 0
I20260925 15:24:29.364087 24541 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:24:29.372512 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 24461
2026-09-25T15:24:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1406 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349610595785-3919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:24:29 dist-test-slave-4bk3 krb5kdc[24560](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:24:29 dist-test-slave-4bk3 krb5kdc[24560](info): set up 2 sockets
Sep 25 15:24:29 dist-test-slave-4bk3 krb5kdc[24560](info): commencing operation
krb5kdc: starting...
W20260925 15:24:30.453047 3919 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Sep 25 15:24:30 dist-test-slave-4bk3 krb5kdc[24560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:24:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:24:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349610595785-3919-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349610595785-3919-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:24:30.588676 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44937
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349610595785-3919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:24:30.675525 24573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:30.675748 24573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:30.675774 24573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:30.677088 24573 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:24:30.677146 24573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:30.677160 24573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:30.677173 24573 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:24:30.677187 24573 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:24:30.678838 24573 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349610595785-3919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349610595785-3919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:24:30.679140 24573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:30.679400 24573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 15:24:30.685495 24573 server_base.cc:1034] running on GCE node
W20260925 15:24:30.685425 24581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:30.685454 24578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:30.685425 24579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:30.686117 24573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:30.686487 24573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:30.687722 24573 hybrid_clock.cc:648] HybridClock initialized: now 1790349870687691 us; error 38 us; skew 500 ppm
Sep 25 15:24:30 dist-test-slave-4bk3 krb5kdc[24560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349870, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:24:30.696646 24573 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:24:30.697299 24573 webserver.cc:460] Webserver started at http://127.0.0.1:35301/ using document root <none> and password file <none>
I20260925 15:24:30.697570 24573 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:30.697645 24573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:30.697810 24573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:30.698967 24573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349610595785-3919-0/minicluster-data/master-0/data/instance:
uuid: "7c1a4ca2bf1d4487ae6e096c5c42fe16"
format_stamp: "Formatted at 2026-09-25 15:24:30 on dist-test-slave-4bk3"
server_key: "3262db3e424cc551461a50132b5b60a0"
server_key_iv: "a070ae83810874f3f84bdd0c62a111be"
server_key_version: "encryptionkey@0"
I20260925 15:24:30.699363 24573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349610595785-3919-0/minicluster-data/master-0/wal/instance:
uuid: "7c1a4ca2bf1d4487ae6e096c5c42fe16"
format_stamp: "Formatted at 2026-09-25 15:24:30 on dist-test-slave-4bk3"
server_key: "3262db3e424cc551461a50132b5b60a0"
server_key_iv: "a070ae83810874f3f84bdd0c62a111be"
server_key_version: "encryptionkey@0"
I20260925 15:24:30.701237 24573 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260925 15:24:30.702276 24588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:30.702620 24573 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:24:30.702766 24573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349610595785-3919-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349610595785-3919-0/minicluster-data/master-0/wal
uuid: "7c1a4ca2bf1d4487ae6e096c5c42fe16"
format_stamp: "Formatted at 2026-09-25 15:24:30 on dist-test-slave-4bk3"
server_key: "3262db3e424cc551461a50132b5b60a0"
server_key_iv: "a070ae83810874f3f84bdd0c62a111be"
server_key_version: "encryptionkey@0"
I20260925 15:24:30.702862 24573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349610595785-3919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349610595785-3919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349610595785-3919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:30.717738 24573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:30.720688 24573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:30.720913 24573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:30.725871 24573 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44937
I20260925 15:24:30.725971 24640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44937 every 8 connection(s)
I20260925 15:24:30.726356 24573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
I20260925 15:24:30.727514 24641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:30.729950 24641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c1a4ca2bf1d4487ae6e096c5c42fe16: Bootstrap starting.
I20260925 15:24:30.730547 24641 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c1a4ca2bf1d4487ae6e096c5c42fe16: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:30.730822 24641 log.cc:824] T 00000000000000000000000000000000 P 7c1a4ca2bf1d4487ae6e096c5c42fe16: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:30.731755 24641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c1a4ca2bf1d4487ae6e096c5c42fe16: No bootstrap required, opened a new log
I20260925 15:24:30.733594 24641 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c1a4ca2bf1d4487ae6e096c5c42fe16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c1a4ca2bf1d4487ae6e096c5c42fe16" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44937 } }
I20260925 15:24:30.733768 24641 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c1a4ca2bf1d4487ae6e096c5c42fe16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:24:30.733803 24641 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c1a4ca2bf1d4487ae6e096c5c42fe16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c1a4ca2bf1d4487ae6e096c5c42fe16, State: Initialized, Role: FOLLOWER
I20260925 15:24:30.733920 24641 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c1a4ca2bf1d4487ae6e096c5c42fe16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c1a4ca2bf1d4487ae6e096c5c42fe16" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44937 } }
I20260925 15:24:30.733983 24641 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c1a4ca2bf1d4487ae6e096c5c42fe16 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:24:30.734006 24641 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c1a4ca2bf1d4487ae6e096c5c42fe16 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:24:30.734037 24641 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c1a4ca2bf1d4487ae6e096c5c42fe16 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:30.734920 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 24573
I20260925 15:24:30.735059 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790349610595785-3919-0/minicluster-data/master-0/wal/instance
I20260925 15:24:30.735224 24641 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c1a4ca2bf1d4487ae6e096c5c42fe16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c1a4ca2bf1d4487ae6e096c5c42fe16" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44937 } }
I20260925 15:24:30.735285 3919 external_mini_cluster.cc:1428] Setting key 1848f1146866ef7b6c307a3901714a8a
I20260925 15:24:30.735354 24641 leader_election.cc:302] T 00000000000000000000000000000000 P 7c1a4ca2bf1d4487ae6e096c5c42fe16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c1a4ca2bf1d4487ae6e096c5c42fe16; no voters:
I20260925 15:24:30.735602 24641 leader_election.cc:288] T 00000000000000000000000000000000 P 7c1a4ca2bf1d4487ae6e096c5c42fe16 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:24:30.735637 24644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c1a4ca2bf1d4487ae6e096c5c42fe16 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:24:30.735764 24644 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c1a4ca2bf1d4487ae6e096c5c42fe16 [term 1 LEADER]: Becoming Leader. State: Replica: 7c1a4ca2bf1d4487ae6e096c5c42fe16, State: Running, Role: LEADER
I20260925 15:24:30.735929 24641 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c1a4ca2bf1d4487ae6e096c5c42fe16 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:24:30.735955 24644 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c1a4ca2bf1d4487ae6e096c5c42fe16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c1a4ca2bf1d4487ae6e096c5c42fe16" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44937 } }
I20260925 15:24:30.736496 24645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c1a4ca2bf1d4487ae6e096c5c42fe16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c1a4ca2bf1d4487ae6e096c5c42fe16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c1a4ca2bf1d4487ae6e096c5c42fe16" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44937 } } }
I20260925 15:24:30.736590 24645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c1a4ca2bf1d4487ae6e096c5c42fe16 [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:30.736828 24646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c1a4ca2bf1d4487ae6e096c5c42fe16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c1a4ca2bf1d4487ae6e096c5c42fe16. Latest consensus state: current_term: 1 leader_uuid: "7c1a4ca2bf1d4487ae6e096c5c42fe16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c1a4ca2bf1d4487ae6e096c5c42fe16" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44937 } } }
I20260925 15:24:30.736959 24646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c1a4ca2bf1d4487ae6e096c5c42fe16 [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:30.737149 24654 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:24:30.737917 24654 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:24:30.739681 24654 catalog_manager.cc:1347] Generated new cluster ID: a684e0db38a542cb8b4f019946448cfc
I20260925 15:24:30.739741 24654 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:24:30.742982 24648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:30.736013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33109 (local address 127.0.0.1:44937)
0925 15:24:30.736180 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:30.736190 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:30.736226 (+ 36us) server_negotiation.cc:407] Connection header received
0925 15:24:30.736398 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:30.736405 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:30.736483 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:30.736635 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:30.737004 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:30.738576 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:30.740376 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:30.740986 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:30.741092 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:30.742209 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:30.742213 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:30.742213 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:24:30.742445 (+ 232us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:30.742487 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:24:30.742502 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:30.742565 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:30.742676 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 15:24:30.742750 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":69,"threads_started":1}
I20260925 15:24:30.744722 3919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:24:30.750007 24648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:30.746308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42831 (local address 127.0.0.1:44937)
0925 15:24:30.746347 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:30.746349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:30.746468 (+ 119us) server_negotiation.cc:407] Connection header received
0925 15:24:30.746558 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:30.746558 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:30.746570 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:30.746597 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:30.746790 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:30.747536 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:30.748600 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:30.748911 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:30.748951 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:30.749731 (+ 780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:30.749733 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:30.749733 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:24:30.749738 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:30.749753 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:24:30.749758 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:30.749795 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:30.749887 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 15:24:30.749918 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 15:24:30.772375 24654 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:24:30.773181 24654 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:24:30.784890 24654 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c1a4ca2bf1d4487ae6e096c5c42fe16: Generated new TSK 0
I20260925 15:24:30.785171 24654 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:24:30.797271 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 24573
2026-09-25T15:24:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1424 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349610595785-3919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:24:30 dist-test-slave-4bk3 krb5kdc[24672](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:24:30 dist-test-slave-4bk3 krb5kdc[24672](info): set up 2 sockets
Sep 25 15:24:30 dist-test-slave-4bk3 krb5kdc[24672](info): commencing operation
krb5kdc: starting...
W20260925 15:24:31.865015 3919 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Sep 25 15:24:31 dist-test-slave-4bk3 krb5kdc[24672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349871, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:24:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:24:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349610595785-3919-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349610595785-3919-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:24:31.996961 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41755
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349610595785-3919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:24:32.090260 24685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:32.090683 24685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:32.090736 24685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:32.092775 24685 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:24:32.092895 24685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:32.092921 24685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:32.092947 24685 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:24:32.092981 24685 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:24:32.095211 24685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349610595785-3919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349610595785-3919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:24:32.095600 24685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:32.095897 24685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:32.101840 24690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:32.101840 24693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:32.101874 24685 server_base.cc:1034] running on GCE node
W20260925 15:24:32.101907 24691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:32.102675 24685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:32.103155 24685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:32.104439 24685 hybrid_clock.cc:648] HybridClock initialized: now 1790349872104397 us; error 41 us; skew 500 ppm
Sep 25 15:24:32 dist-test-slave-4bk3 krb5kdc[24672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349872, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:24:32.108103 24685 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:24:32.108702 24685 webserver.cc:460] Webserver started at http://127.0.0.1:41657/ using document root <none> and password file <none>
I20260925 15:24:32.108964 24685 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:32.109037 24685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:32.109194 24685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:32.110307 24685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349610595785-3919-0/minicluster-data/master-0/data/instance:
uuid: "1cbf909da92d43439d4d8f16942f37c8"
format_stamp: "Formatted at 2026-09-25 15:24:32 on dist-test-slave-4bk3"
server_key: "30d65a5a146f25f87090f60e74038537"
server_key_iv: "f5ab769dc4c31f2572d97dc267c54ef2"
server_key_version: "encryptionkey@0"
I20260925 15:24:32.110688 24685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349610595785-3919-0/minicluster-data/master-0/wal/instance:
uuid: "1cbf909da92d43439d4d8f16942f37c8"
format_stamp: "Formatted at 2026-09-25 15:24:32 on dist-test-slave-4bk3"
server_key: "30d65a5a146f25f87090f60e74038537"
server_key_iv: "f5ab769dc4c31f2572d97dc267c54ef2"
server_key_version: "encryptionkey@0"
I20260925 15:24:32.112196 24685 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:24:32.113227 24700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:32.113524 24685 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:24:32.113607 24685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349610595785-3919-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349610595785-3919-0/minicluster-data/master-0/wal
uuid: "1cbf909da92d43439d4d8f16942f37c8"
format_stamp: "Formatted at 2026-09-25 15:24:32 on dist-test-slave-4bk3"
server_key: "30d65a5a146f25f87090f60e74038537"
server_key_iv: "f5ab769dc4c31f2572d97dc267c54ef2"
server_key_version: "encryptionkey@0"
I20260925 15:24:32.113705 24685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349610595785-3919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349610595785-3919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349610595785-3919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:32.117522 24685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:32.120496 24685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:32.120672 24685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:32.125068 24685 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41755
I20260925 15:24:32.125101 24752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41755 every 8 connection(s)
I20260925 15:24:32.125461 24685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
I20260925 15:24:32.126530 24753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:32.129032 24753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cbf909da92d43439d4d8f16942f37c8: Bootstrap starting.
I20260925 15:24:32.129658 24753 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1cbf909da92d43439d4d8f16942f37c8: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:32.129957 24753 log.cc:824] T 00000000000000000000000000000000 P 1cbf909da92d43439d4d8f16942f37c8: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:32.131107 24753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cbf909da92d43439d4d8f16942f37c8: No bootstrap required, opened a new log
I20260925 15:24:32.132701 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 24685
I20260925 15:24:32.132800 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790349610595785-3919-0/minicluster-data/master-0/wal/instance
I20260925 15:24:32.133010 3919 external_mini_cluster.cc:1428] Setting key 1afc70703e450fd25abadc245e29af1d
I20260925 15:24:32.134243 24753 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1cbf909da92d43439d4d8f16942f37c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cbf909da92d43439d4d8f16942f37c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41755 } }
I20260925 15:24:32.134467 24753 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1cbf909da92d43439d4d8f16942f37c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:24:32.134508 24753 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1cbf909da92d43439d4d8f16942f37c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbf909da92d43439d4d8f16942f37c8, State: Initialized, Role: FOLLOWER
I20260925 15:24:32.134636 24753 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1cbf909da92d43439d4d8f16942f37c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cbf909da92d43439d4d8f16942f37c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41755 } }
I20260925 15:24:32.134735 24753 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1cbf909da92d43439d4d8f16942f37c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:24:32.134770 24753 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1cbf909da92d43439d4d8f16942f37c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:24:32.134843 24753 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1cbf909da92d43439d4d8f16942f37c8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:32.135944 24757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:32.133782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47099 (local address 127.0.0.1:41755)
0925 15:24:32.134102 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:32.134114 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:32.134164 (+ 50us) server_negotiation.cc:407] Connection header received
0925 15:24:32.134301 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:32.134309 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:32.134342 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:32.134523 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:32.135333 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:32.135338 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:32.135338 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:24:32.135345 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:32.135403 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:24:32.135422 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:32.135517 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:32.135655 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 15:24:32.135702 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":161,"threads_started":1}
I20260925 15:24:32.136152 24753 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1cbf909da92d43439d4d8f16942f37c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cbf909da92d43439d4d8f16942f37c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41755 } }
I20260925 15:24:32.136286 24753 leader_election.cc:302] T 00000000000000000000000000000000 P 1cbf909da92d43439d4d8f16942f37c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1cbf909da92d43439d4d8f16942f37c8; no voters:
I20260925 15:24:32.136502 24753 leader_election.cc:288] T 00000000000000000000000000000000 P 1cbf909da92d43439d4d8f16942f37c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:24:32.136554 24758 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1cbf909da92d43439d4d8f16942f37c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:24:32.136719 24758 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1cbf909da92d43439d4d8f16942f37c8 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbf909da92d43439d4d8f16942f37c8, State: Running, Role: LEADER
I20260925 15:24:32.136772 24753 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1cbf909da92d43439d4d8f16942f37c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:24:32.136883 24758 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1cbf909da92d43439d4d8f16942f37c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cbf909da92d43439d4d8f16942f37c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41755 } }
I20260925 15:24:32.137072 3919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:24:32.137339 24760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cbf909da92d43439d4d8f16942f37c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cbf909da92d43439d4d8f16942f37c8. Latest consensus state: current_term: 1 leader_uuid: "1cbf909da92d43439d4d8f16942f37c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cbf909da92d43439d4d8f16942f37c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41755 } } }
I20260925 15:24:32.137439 24760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cbf909da92d43439d4d8f16942f37c8 [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:32.137331 24759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cbf909da92d43439d4d8f16942f37c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1cbf909da92d43439d4d8f16942f37c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cbf909da92d43439d4d8f16942f37c8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41755 } } }
I20260925 15:24:32.137482 24759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cbf909da92d43439d4d8f16942f37c8 [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:32.137902 24766 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:24:32.138711 24766 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:24:32.141863 24766 catalog_manager.cc:1347] Generated new cluster ID: 219aef4eb9af48ddaaf8af23b63b76e5
I20260925 15:24:32.142150 24766 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:24:32.142472 24757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:32.141055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58861 (local address 127.0.0.1:41755)
0925 15:24:32.141092 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:32.141094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:32.141261 (+ 167us) server_negotiation.cc:407] Connection header received
0925 15:24:32.141327 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:32.141328 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:32.141334 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:32.141424 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:32.142226 (+ 802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:32.142227 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:32.142227 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:24:32.142232 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:32.142250 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:24:32.142254 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:32.142319 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:32.142372 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 15:24:32.142388 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 15:24:32.160900 24766 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:24:32.161139 24766 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:24:32.168445 24766 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1cbf909da92d43439d4d8f16942f37c8: Generated new TSK 0
I20260925 15:24:32.168869 24766 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:24:32.174949 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 24685
2026-09-25T15:24:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1377 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349610595785-3919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:24:32 dist-test-slave-4bk3 krb5kdc[24784](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:24:32 dist-test-slave-4bk3 krb5kdc[24784](info): set up 2 sockets
Sep 25 15:24:32 dist-test-slave-4bk3 krb5kdc[24784](info): commencing operation
krb5kdc: starting...
W20260925 15:24:33.233665 3919 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Sep 25 15:24:33 dist-test-slave-4bk3 krb5kdc[24784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:24:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:24:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349610595785-3919-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349610595785-3919-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:24:33.372745 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39073
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349610595785-3919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39073
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:24:33.466467 24797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:33.466708 24797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:33.466734 24797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:33.468066 24797 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:24:33.468125 24797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:33.468142 24797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:33.468155 24797 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:24:33.468168 24797 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:24:33.469837 24797 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349610595785-3919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39073
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349610595785-3919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:24:33.470127 24797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:33.470419 24797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:33.476639 24803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:33.476620 24802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:33.476742 24797 server_base.cc:1034] running on GCE node
W20260925 15:24:33.476830 24805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:33.477103 24797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:33.477443 24797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:33.478618 24797 hybrid_clock.cc:648] HybridClock initialized: now 1790349873478605 us; error 35 us; skew 500 ppm
Sep 25 15:24:33 dist-test-slave-4bk3 krb5kdc[24784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349873, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:24:33.481839 24797 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:24:33.482357 24797 webserver.cc:460] Webserver started at http://127.0.0.1:37025/ using document root <none> and password file <none>
I20260925 15:24:33.482611 24797 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:33.482664 24797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:33.482800 24797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:33.483947 24797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349610595785-3919-0/minicluster-data/master-0/data/instance:
uuid: "e2024c13b7fc4cc88744f644b6e7cc4f"
format_stamp: "Formatted at 2026-09-25 15:24:33 on dist-test-slave-4bk3"
server_key: "c32828eb3c30c5c2e5e6b60cf64d19b9"
server_key_iv: "895da824296e9d356808feb7bd2d64ad"
server_key_version: "encryptionkey@0"
I20260925 15:24:33.484373 24797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349610595785-3919-0/minicluster-data/master-0/wal/instance:
uuid: "e2024c13b7fc4cc88744f644b6e7cc4f"
format_stamp: "Formatted at 2026-09-25 15:24:33 on dist-test-slave-4bk3"
server_key: "c32828eb3c30c5c2e5e6b60cf64d19b9"
server_key_iv: "895da824296e9d356808feb7bd2d64ad"
server_key_version: "encryptionkey@0"
I20260925 15:24:33.486217 24797 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260925 15:24:33.487383 24812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:33.487767 24797 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:24:33.487876 24797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349610595785-3919-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349610595785-3919-0/minicluster-data/master-0/wal
uuid: "e2024c13b7fc4cc88744f644b6e7cc4f"
format_stamp: "Formatted at 2026-09-25 15:24:33 on dist-test-slave-4bk3"
server_key: "c32828eb3c30c5c2e5e6b60cf64d19b9"
server_key_iv: "895da824296e9d356808feb7bd2d64ad"
server_key_version: "encryptionkey@0"
I20260925 15:24:33.487998 24797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349610595785-3919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349610595785-3919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349610595785-3919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:33.497259 24797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:33.500888 24797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:33.501024 24797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:33.505410 24797 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39073
I20260925 15:24:33.505470 24864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39073 every 8 connection(s)
I20260925 15:24:33.505856 24797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
I20260925 15:24:33.506856 24865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:33.508596 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 24797
I20260925 15:24:33.508700 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790349610595785-3919-0/minicluster-data/master-0/wal/instance
I20260925 15:24:33.508934 3919 external_mini_cluster.cc:1428] Setting key e90202c1161aefe8cfcc9c26dc673393
I20260925 15:24:33.509601 24865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2024c13b7fc4cc88744f644b6e7cc4f: Bootstrap starting.
I20260925 15:24:33.510224 24865 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2024c13b7fc4cc88744f644b6e7cc4f: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:33.510600 24865 log.cc:824] T 00000000000000000000000000000000 P e2024c13b7fc4cc88744f644b6e7cc4f: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:33.511413 24865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2024c13b7fc4cc88744f644b6e7cc4f: No bootstrap required, opened a new log
I20260925 15:24:33.513595 24865 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2024c13b7fc4cc88744f644b6e7cc4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2024c13b7fc4cc88744f644b6e7cc4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39073 } }
I20260925 15:24:33.513803 24865 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2024c13b7fc4cc88744f644b6e7cc4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:24:33.513854 24865 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2024c13b7fc4cc88744f644b6e7cc4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2024c13b7fc4cc88744f644b6e7cc4f, State: Initialized, Role: FOLLOWER
I20260925 15:24:33.514031 24865 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2024c13b7fc4cc88744f644b6e7cc4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2024c13b7fc4cc88744f644b6e7cc4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39073 } }
I20260925 15:24:33.514101 24865 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2024c13b7fc4cc88744f644b6e7cc4f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:24:33.514145 24865 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2024c13b7fc4cc88744f644b6e7cc4f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:24:33.514197 24865 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2024c13b7fc4cc88744f644b6e7cc4f [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:33.514997 24865 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2024c13b7fc4cc88744f644b6e7cc4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2024c13b7fc4cc88744f644b6e7cc4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39073 } }
I20260925 15:24:33.515264 24865 leader_election.cc:302] T 00000000000000000000000000000000 P e2024c13b7fc4cc88744f644b6e7cc4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2024c13b7fc4cc88744f644b6e7cc4f; no voters:
I20260925 15:24:33.515519 24865 leader_election.cc:288] T 00000000000000000000000000000000 P e2024c13b7fc4cc88744f644b6e7cc4f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:24:33.515554 24870 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2024c13b7fc4cc88744f644b6e7cc4f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:24:33.515674 24870 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2024c13b7fc4cc88744f644b6e7cc4f [term 1 LEADER]: Becoming Leader. State: Replica: e2024c13b7fc4cc88744f644b6e7cc4f, State: Running, Role: LEADER
I20260925 15:24:33.515831 24865 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2024c13b7fc4cc88744f644b6e7cc4f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:24:33.515811 24870 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2024c13b7fc4cc88744f644b6e7cc4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2024c13b7fc4cc88744f644b6e7cc4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39073 } }
I20260925 15:24:33.516397 24871 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2024c13b7fc4cc88744f644b6e7cc4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2024c13b7fc4cc88744f644b6e7cc4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2024c13b7fc4cc88744f644b6e7cc4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39073 } } }
I20260925 15:24:33.516518 24871 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2024c13b7fc4cc88744f644b6e7cc4f [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:33.516387 24870 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2024c13b7fc4cc88744f644b6e7cc4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2024c13b7fc4cc88744f644b6e7cc4f. Latest consensus state: current_term: 1 leader_uuid: "e2024c13b7fc4cc88744f644b6e7cc4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2024c13b7fc4cc88744f644b6e7cc4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39073 } } }
I20260925 15:24:33.516834 24870 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2024c13b7fc4cc88744f644b6e7cc4f [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:33.516993 24878 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 25 15:24:33 dist-test-slave-4bk3 krb5kdc[24784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 15:24:33.517799 24878 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:24:33.520366 24878 catalog_manager.cc:1347] Generated new cluster ID: 2baec8d64b11469ca8f8ed1e2900300d
I20260925 15:24:33.520427 24878 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:24:33.522917 24868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:33.509507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.79:41649 (local address 127.0.0.1:39073)
0925 15:24:33.509708 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:33.509719 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:33.509758 (+ 39us) server_negotiation.cc:407] Connection header received
0925 15:24:33.510214 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:33.510221 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:33.510271 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:33.510448 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:33.511271 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:33.512679 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:33.513869 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:33.514515 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:33.514612 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:33.518085 (+ 3473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:33.518089 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:33.518090 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:33.518708 (+ 618us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:33.521024 (+ 2316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:33.521598 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:33.521600 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:33.521601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:33.521710 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:33.521976 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:33.521977 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:33.521977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:33.522284 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:33.522447 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:33.522603 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 15:24:33.522699 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":89,"threads_started":1}
I20260925 15:24:33.525496 3919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:24:33.534584 24868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:33.526964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.79:50603 (local address 127.0.0.1:39073)
0925 15:24:33.527005 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:33.527007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:33.527254 (+ 247us) server_negotiation.cc:407] Connection header received
0925 15:24:33.527338 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:33.527339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:33.527349 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:33.527376 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:33.528037 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:33.528807 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:33.529790 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:33.530112 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:33.530149 (+ 37us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:33.531702 (+ 1553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:33.531704 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:33.531704 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:33.531711 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:33.533307 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:33.533758 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:33.533759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:33.533759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:33.533809 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:33.534024 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:33.534025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:33.534025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:33.534144 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:33.534214 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:33.534419 (+ 205us) server_negotiation.cc:299] Negotiation successful
0925 15:24:33.534486 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 15:24:33.537562 24878 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:24:33.538311 24878 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:24:33.541932 24878 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2024c13b7fc4cc88744f644b6e7cc4f: Generated new TSK 0
I20260925 15:24:33.542140 24878 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:24:33.546914 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 24797
W20260925 15:24:33.550930 24782 connection.cc:535] client connection to 127.0.0.1:39073 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39073): Connection reset by peer (error 104)
2026-09-25T15:24:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1371 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349610595785-3919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:24:33 dist-test-slave-4bk3 krb5kdc[24896](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:24:33 dist-test-slave-4bk3 krb5kdc[24896](info): set up 2 sockets
Sep 25 15:24:33 dist-test-slave-4bk3 krb5kdc[24896](info): commencing operation
krb5kdc: starting...
W20260925 15:24:34.616683 3919 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Sep 25 15:24:34 dist-test-slave-4bk3 krb5kdc[24896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:24:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:24:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349610595785-3919-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349610595785-3919-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:24:34.756958 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46079
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349610595785-3919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46079
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:24:34.855232 24909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:34.855497 24909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:34.855526 24909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:34.857049 24909 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:24:34.857105 24909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:34.857120 24909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:34.857133 24909 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:24:34.857146 24909 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:24:34.858829 24909 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349610595785-3919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349610595785-3919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:24:34.859068 24909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:34.859267 24909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 15:24:34.865494 24909 server_base.cc:1034] running on GCE node
W20260925 15:24:34.865442 24914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:34.865442 24917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:34.865442 24915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:34.866170 24909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:34.866518 24909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:34.867699 24909 hybrid_clock.cc:648] HybridClock initialized: now 1790349874867686 us; error 33 us; skew 500 ppm
Sep 25 15:24:34 dist-test-slave-4bk3 krb5kdc[24896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:24:34.870579 24909 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:24:34.871155 24909 webserver.cc:460] Webserver started at http://127.0.0.1:45635/ using document root <none> and password file <none>
I20260925 15:24:34.871461 24909 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:34.871538 24909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:34.871693 24909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:34.872959 24909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349610595785-3919-0/minicluster-data/master-0/data/instance:
uuid: "81f7bc0bc71b4715a8a81d6a337fa4d7"
format_stamp: "Formatted at 2026-09-25 15:24:34 on dist-test-slave-4bk3"
server_key: "e12c9d4ae44899c641ec9d056de0cb23"
server_key_iv: "10f67d7d12744e8390e88df72bfe668d"
server_key_version: "encryptionkey@0"
I20260925 15:24:34.873361 24909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349610595785-3919-0/minicluster-data/master-0/wal/instance:
uuid: "81f7bc0bc71b4715a8a81d6a337fa4d7"
format_stamp: "Formatted at 2026-09-25 15:24:34 on dist-test-slave-4bk3"
server_key: "e12c9d4ae44899c641ec9d056de0cb23"
server_key_iv: "10f67d7d12744e8390e88df72bfe668d"
server_key_version: "encryptionkey@0"
I20260925 15:24:34.874886 24909 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:24:34.875937 24924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:34.876155 24909 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:24:34.876302 24909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349610595785-3919-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349610595785-3919-0/minicluster-data/master-0/wal
uuid: "81f7bc0bc71b4715a8a81d6a337fa4d7"
format_stamp: "Formatted at 2026-09-25 15:24:34 on dist-test-slave-4bk3"
server_key: "e12c9d4ae44899c641ec9d056de0cb23"
server_key_iv: "10f67d7d12744e8390e88df72bfe668d"
server_key_version: "encryptionkey@0"
I20260925 15:24:34.876451 24909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349610595785-3919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349610595785-3919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349610595785-3919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:34.886650 24909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:34.889286 24909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:34.889468 24909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:34.894376 24909 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46079
I20260925 15:24:34.894423 24976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46079 every 8 connection(s)
I20260925 15:24:34.894814 24909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
I20260925 15:24:34.896534 24977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:34.899189 24977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81f7bc0bc71b4715a8a81d6a337fa4d7: Bootstrap starting.
I20260925 15:24:34.899861 24977 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81f7bc0bc71b4715a8a81d6a337fa4d7: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:34.900170 24977 log.cc:824] T 00000000000000000000000000000000 P 81f7bc0bc71b4715a8a81d6a337fa4d7: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:34.901005 24977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81f7bc0bc71b4715a8a81d6a337fa4d7: No bootstrap required, opened a new log
I20260925 15:24:34.902761 24977 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81f7bc0bc71b4715a8a81d6a337fa4d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81f7bc0bc71b4715a8a81d6a337fa4d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46079 } }
I20260925 15:24:34.902930 24977 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81f7bc0bc71b4715a8a81d6a337fa4d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:24:34.902956 24977 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81f7bc0bc71b4715a8a81d6a337fa4d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81f7bc0bc71b4715a8a81d6a337fa4d7, State: Initialized, Role: FOLLOWER
I20260925 15:24:34.903127 24977 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81f7bc0bc71b4715a8a81d6a337fa4d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81f7bc0bc71b4715a8a81d6a337fa4d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46079 } }
I20260925 15:24:34.903194 24977 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81f7bc0bc71b4715a8a81d6a337fa4d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:24:34.903218 24977 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81f7bc0bc71b4715a8a81d6a337fa4d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:24:34.903270 24977 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81f7bc0bc71b4715a8a81d6a337fa4d7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:34.903505 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 24909
I20260925 15:24:34.903635 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790349610595785-3919-0/minicluster-data/master-0/wal/instance
I20260925 15:24:34.903855 3919 external_mini_cluster.cc:1428] Setting key cb06b760ce62b3ec6bc6b72f47cae109
I20260925 15:24:34.904021 24977 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81f7bc0bc71b4715a8a81d6a337fa4d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81f7bc0bc71b4715a8a81d6a337fa4d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46079 } }
I20260925 15:24:34.904158 24977 leader_election.cc:302] T 00000000000000000000000000000000 P 81f7bc0bc71b4715a8a81d6a337fa4d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81f7bc0bc71b4715a8a81d6a337fa4d7; no voters:
I20260925 15:24:34.904400 24977 leader_election.cc:288] T 00000000000000000000000000000000 P 81f7bc0bc71b4715a8a81d6a337fa4d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:24:34.904428 24980 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81f7bc0bc71b4715a8a81d6a337fa4d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:24:34.904647 24980 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81f7bc0bc71b4715a8a81d6a337fa4d7 [term 1 LEADER]: Becoming Leader. State: Replica: 81f7bc0bc71b4715a8a81d6a337fa4d7, State: Running, Role: LEADER
I20260925 15:24:34.904742 24980 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81f7bc0bc71b4715a8a81d6a337fa4d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81f7bc0bc71b4715a8a81d6a337fa4d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46079 } }
I20260925 15:24:34.904780 24977 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81f7bc0bc71b4715a8a81d6a337fa4d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:24:34.905170 24980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81f7bc0bc71b4715a8a81d6a337fa4d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81f7bc0bc71b4715a8a81d6a337fa4d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81f7bc0bc71b4715a8a81d6a337fa4d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46079 } } }
I20260925 15:24:34.905328 24980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81f7bc0bc71b4715a8a81d6a337fa4d7 [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:34.905498 24980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81f7bc0bc71b4715a8a81d6a337fa4d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81f7bc0bc71b4715a8a81d6a337fa4d7. Latest consensus state: current_term: 1 leader_uuid: "81f7bc0bc71b4715a8a81d6a337fa4d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81f7bc0bc71b4715a8a81d6a337fa4d7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46079 } } }
I20260925 15:24:34.905613 24980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81f7bc0bc71b4715a8a81d6a337fa4d7 [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:34.906756 24986 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:24:34.908083 24986 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:24:34.909945 24986 catalog_manager.cc:1347] Generated new cluster ID: d7475bee7a6e41f5b254d43ca85d9c3d
I20260925 15:24:34.910002 24986 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:24:34.910795 24984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:34.904665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.79:47769 (local address 127.0.0.1:46079)
0925 15:24:34.904831 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:34.904840 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:34.904889 (+ 49us) server_negotiation.cc:407] Connection header received
0925 15:24:34.905026 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:34.905032 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:34.905081 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:34.905486 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:34.905588 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:34.907008 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:34.907886 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:34.908728 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:34.908890 (+ 162us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:34.910106 (+ 1216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:34.910110 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:34.910111 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:24:34.910311 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:34.910348 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:24:34.910360 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:34.910462 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:34.910543 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 15:24:34.910578 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":81,"threads_started":1}
I20260925 15:24:34.911726 3919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:24:34.914593 24986 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:24:34.915390 24986 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:24:34.919029 24984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:34.913161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.79:33975 (local address 127.0.0.1:46079)
0925 15:24:34.913202 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:34.913204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:34.914196 (+ 992us) server_negotiation.cc:407] Connection header received
0925 15:24:34.914325 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:34.914326 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:34.914344 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:34.914413 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:34.914686 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:34.915827 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:34.916904 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:34.917141 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:34.917223 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:34.918465 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:34.918472 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:34.918472 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:24:34.918480 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:34.918502 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:24:34.918508 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:34.918587 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:34.918893 (+ 306us) server_negotiation.cc:299] Negotiation successful
0925 15:24:34.918912 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":20}
I20260925 15:24:34.923712 24986 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81f7bc0bc71b4715a8a81d6a337fa4d7: Generated new TSK 0
I20260925 15:24:34.924221 24986 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:24:34.927717 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 24909
W20260925 15:24:34.932504 24894 connection.cc:535] client connection to 127.0.0.1:46079 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46079): Connection reset by peer (error 104)
2026-09-25T15:24:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1381 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349610595785-3919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:24:34 dist-test-slave-4bk3 krb5kdc[25008](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:24:34 dist-test-slave-4bk3 krb5kdc[25008](info): set up 2 sockets
Sep 25 15:24:34 dist-test-slave-4bk3 krb5kdc[25008](info): commencing operation
krb5kdc: starting...
W20260925 15:24:36.009974 3919 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Sep 25 15:24:36 dist-test-slave-4bk3 krb5kdc[25008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349876, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:24:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:24:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349610595785-3919-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349610595785-3919-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:24:36.144701 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36155
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349610595785-3919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:24:36.232708 25021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:36.232997 25021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:36.233038 25021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:36.234360 25021 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:24:36.234452 25021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:36.234477 25021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:36.234508 25021 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:24:36.234549 25021 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:24:36.236280 25021 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349610595785-3919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349610595785-3919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:24:36.236711 25021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:36.236964 25021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:36.242161 25026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:36.242318 25021 server_base.cc:1034] running on GCE node
W20260925 15:24:36.242367 25027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:36.242506 25029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:36.242801 25021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:36.243068 25021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:36.244310 25021 hybrid_clock.cc:648] HybridClock initialized: now 1790349876244290 us; error 35 us; skew 500 ppm
Sep 25 15:24:36 dist-test-slave-4bk3 krb5kdc[25008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349876, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:24:36.250698 25021 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:24:36.251268 25021 webserver.cc:460] Webserver started at http://127.0.0.1:42669/ using document root <none> and password file <none>
I20260925 15:24:36.251525 25021 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:36.251572 25021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:36.251716 25021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:36.252871 25021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349610595785-3919-0/minicluster-data/master-0/data/instance:
uuid: "ab01eb4ddd0445f5bc3b95e722824c92"
format_stamp: "Formatted at 2026-09-25 15:24:36 on dist-test-slave-4bk3"
server_key: "55ac19ed5e62ce4f9400aa3864e69619"
server_key_iv: "a9e1c381028ce03b72ae296ae87464ec"
server_key_version: "encryptionkey@0"
I20260925 15:24:36.253237 25021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349610595785-3919-0/minicluster-data/master-0/wal/instance:
uuid: "ab01eb4ddd0445f5bc3b95e722824c92"
format_stamp: "Formatted at 2026-09-25 15:24:36 on dist-test-slave-4bk3"
server_key: "55ac19ed5e62ce4f9400aa3864e69619"
server_key_iv: "a9e1c381028ce03b72ae296ae87464ec"
server_key_version: "encryptionkey@0"
I20260925 15:24:36.254627 25021 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:24:36.255437 25036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:36.255705 25021 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:24:36.255812 25021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349610595785-3919-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349610595785-3919-0/minicluster-data/master-0/wal
uuid: "ab01eb4ddd0445f5bc3b95e722824c92"
format_stamp: "Formatted at 2026-09-25 15:24:36 on dist-test-slave-4bk3"
server_key: "55ac19ed5e62ce4f9400aa3864e69619"
server_key_iv: "a9e1c381028ce03b72ae296ae87464ec"
server_key_version: "encryptionkey@0"
I20260925 15:24:36.255898 25021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349610595785-3919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349610595785-3919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349610595785-3919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:36.259948 25021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:36.262862 25021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:36.263067 25021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:36.268875 25021 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36155
I20260925 15:24:36.268972 25088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36155 every 8 connection(s)
I20260925 15:24:36.269421 25021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
I20260925 15:24:36.270460 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 25021
I20260925 15:24:36.270558 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790349610595785-3919-0/minicluster-data/master-0/wal/instance
I20260925 15:24:36.270634 25089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:36.270807 3919 external_mini_cluster.cc:1428] Setting key 7f8633c77448e465be2a80124eccbc33
I20260925 15:24:36.273509 25092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:36.271406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.79:41157 (local address 127.0.0.1:36155)
0925 15:24:36.271640 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:36.271648 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:36.271675 (+ 27us) server_negotiation.cc:407] Connection header received
0925 15:24:36.272130 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:36.272138 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:36.272164 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:36.272389 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:36.273070 (+ 681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:36.273079 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:36.273079 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:24:36.273084 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:36.273119 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:24:36.273131 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:36.273211 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:36.273279 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 15:24:36.273349 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":99,"threads_started":1}
I20260925 15:24:36.274369 25089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab01eb4ddd0445f5bc3b95e722824c92: Bootstrap starting.
I20260925 15:24:36.274626 3919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:24:36.275157 25089 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab01eb4ddd0445f5bc3b95e722824c92: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:36.275504 25089 log.cc:824] T 00000000000000000000000000000000 P ab01eb4ddd0445f5bc3b95e722824c92: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:36.276806 25089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab01eb4ddd0445f5bc3b95e722824c92: No bootstrap required, opened a new log
I20260925 15:24:36.277534 25092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:36.276406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.79:47637 (local address 127.0.0.1:36155)
0925 15:24:36.276463 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:36.276465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:36.276534 (+ 69us) server_negotiation.cc:407] Connection header received
0925 15:24:36.276567 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:36.276568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:36.276574 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:36.276644 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:36.277299 (+ 655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:36.277300 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:36.277301 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:24:36.277305 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:36.277321 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:24:36.277326 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:36.277396 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:36.277456 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 15:24:36.277472 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 15:24:36.279069 25089 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab01eb4ddd0445f5bc3b95e722824c92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab01eb4ddd0445f5bc3b95e722824c92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36155 } }
I20260925 15:24:36.279225 25089 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab01eb4ddd0445f5bc3b95e722824c92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:24:36.279248 25089 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab01eb4ddd0445f5bc3b95e722824c92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab01eb4ddd0445f5bc3b95e722824c92, State: Initialized, Role: FOLLOWER
I20260925 15:24:36.279340 25089 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab01eb4ddd0445f5bc3b95e722824c92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab01eb4ddd0445f5bc3b95e722824c92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36155 } }
I20260925 15:24:36.279443 25089 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab01eb4ddd0445f5bc3b95e722824c92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:24:36.279508 25089 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab01eb4ddd0445f5bc3b95e722824c92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:24:36.279584 25089 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab01eb4ddd0445f5bc3b95e722824c92 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:36.280203 25089 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab01eb4ddd0445f5bc3b95e722824c92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab01eb4ddd0445f5bc3b95e722824c92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36155 } }
I20260925 15:24:36.280296 25089 leader_election.cc:302] T 00000000000000000000000000000000 P ab01eb4ddd0445f5bc3b95e722824c92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab01eb4ddd0445f5bc3b95e722824c92; no voters:
I20260925 15:24:36.280543 25089 leader_election.cc:288] T 00000000000000000000000000000000 P ab01eb4ddd0445f5bc3b95e722824c92 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:24:36.280623 25099 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab01eb4ddd0445f5bc3b95e722824c92 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:24:36.280804 25089 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab01eb4ddd0445f5bc3b95e722824c92 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:24:36.280874 25099 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab01eb4ddd0445f5bc3b95e722824c92 [term 1 LEADER]: Becoming Leader. State: Replica: ab01eb4ddd0445f5bc3b95e722824c92, State: Running, Role: LEADER
I20260925 15:24:36.280973 25099 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab01eb4ddd0445f5bc3b95e722824c92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab01eb4ddd0445f5bc3b95e722824c92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36155 } }
I20260925 15:24:36.281361 25099 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab01eb4ddd0445f5bc3b95e722824c92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab01eb4ddd0445f5bc3b95e722824c92. Latest consensus state: current_term: 1 leader_uuid: "ab01eb4ddd0445f5bc3b95e722824c92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab01eb4ddd0445f5bc3b95e722824c92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36155 } } }
I20260925 15:24:36.281440 25099 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab01eb4ddd0445f5bc3b95e722824c92 [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:36.281466 25100 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab01eb4ddd0445f5bc3b95e722824c92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab01eb4ddd0445f5bc3b95e722824c92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab01eb4ddd0445f5bc3b95e722824c92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36155 } } }
I20260925 15:24:36.281561 25100 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab01eb4ddd0445f5bc3b95e722824c92 [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:36.281718 25105 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:24:36.282280 25105 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:24:36.283978 25105 catalog_manager.cc:1347] Generated new cluster ID: 3fb536ba446542a793725cfbbecd7a41
I20260925 15:24:36.284049 25105 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:24:36.308822 25105 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:24:36.309039 25105 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:24:36.313457 25105 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab01eb4ddd0445f5bc3b95e722824c92: Generated new TSK 0
I20260925 15:24:36.313714 25105 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:24:36.321671 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 25021
2026-09-25T15:24:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1393 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349610595785-3919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:24:36 dist-test-slave-4bk3 krb5kdc[25120](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:24:36 dist-test-slave-4bk3 krb5kdc[25120](info): set up 2 sockets
Sep 25 15:24:36 dist-test-slave-4bk3 krb5kdc[25120](info): commencing operation
krb5kdc: starting...
W20260925 15:24:37.409847 3919 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Sep 25 15:24:37 dist-test-slave-4bk3 krb5kdc[25120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:24:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:24:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349610595785-3919-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349610595785-3919-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:24:37.553649 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46535
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349610595785-3919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46535
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:24:37.647457 25133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:37.647763 25133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:37.647809 25133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:37.649539 25133 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:24:37.649648 25133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:37.649675 25133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:37.649708 25133 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:24:37.649739 25133 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:24:37.651649 25133 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349610595785-3919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46535
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349610595785-3919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:24:37.652000 25133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:37.652257 25133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:37.657683 25141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:37.657683 25139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:37.657683 25138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:37.657886 25133 server_base.cc:1034] running on GCE node
I20260925 15:24:37.658226 25133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:37.658593 25133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:37.659782 25133 hybrid_clock.cc:648] HybridClock initialized: now 1790349877659761 us; error 48 us; skew 500 ppm
Sep 25 15:24:37 dist-test-slave-4bk3 krb5kdc[25120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:24:37.662629 25133 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:24:37.663609 25133 webserver.cc:460] Webserver started at http://127.0.0.1:46541/ using document root <none> and password file <none>
I20260925 15:24:37.663976 25133 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:37.664074 25133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:37.664259 25133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:37.665642 25133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349610595785-3919-0/minicluster-data/master-0/data/instance:
uuid: "211b80de3926480f9404708d45bfc84c"
format_stamp: "Formatted at 2026-09-25 15:24:37 on dist-test-slave-4bk3"
server_key: "ab3b7260b074ce6a9fb48299c9c68e9c"
server_key_iv: "d6d2ab670968a8a47cc1a7c3dcb67fc6"
server_key_version: "encryptionkey@0"
I20260925 15:24:37.666095 25133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349610595785-3919-0/minicluster-data/master-0/wal/instance:
uuid: "211b80de3926480f9404708d45bfc84c"
format_stamp: "Formatted at 2026-09-25 15:24:37 on dist-test-slave-4bk3"
server_key: "ab3b7260b074ce6a9fb48299c9c68e9c"
server_key_iv: "d6d2ab670968a8a47cc1a7c3dcb67fc6"
server_key_version: "encryptionkey@0"
I20260925 15:24:37.668081 25133 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260925 15:24:37.669220 25148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:37.669502 25133 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:24:37.669648 25133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349610595785-3919-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349610595785-3919-0/minicluster-data/master-0/wal
uuid: "211b80de3926480f9404708d45bfc84c"
format_stamp: "Formatted at 2026-09-25 15:24:37 on dist-test-slave-4bk3"
server_key: "ab3b7260b074ce6a9fb48299c9c68e9c"
server_key_iv: "d6d2ab670968a8a47cc1a7c3dcb67fc6"
server_key_version: "encryptionkey@0"
I20260925 15:24:37.669765 25133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349610595785-3919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349610595785-3919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349610595785-3919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:37.687352 25133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:37.689960 25133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:37.690146 25133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:37.694950 25133 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46535
I20260925 15:24:37.694996 25200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46535 every 8 connection(s)
I20260925 15:24:37.695385 25133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
I20260925 15:24:37.696838 25201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:37.700064 25201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 211b80de3926480f9404708d45bfc84c: Bootstrap starting.
I20260925 15:24:37.700268 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 25133
I20260925 15:24:37.700438 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790349610595785-3919-0/minicluster-data/master-0/wal/instance
I20260925 15:24:37.700634 3919 external_mini_cluster.cc:1428] Setting key 8111584a9a5ee440b59ea8b3e3eca4b6
I20260925 15:24:37.700907 25201 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 211b80de3926480f9404708d45bfc84c: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:37.701297 25201 log.cc:824] T 00000000000000000000000000000000 P 211b80de3926480f9404708d45bfc84c: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:37.702116 25201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 211b80de3926480f9404708d45bfc84c: No bootstrap required, opened a new log
I20260925 15:24:37.703938 25201 raft_consensus.cc:348] T 00000000000000000000000000000000 P 211b80de3926480f9404708d45bfc84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211b80de3926480f9404708d45bfc84c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46535 } }
I20260925 15:24:37.704136 25201 raft_consensus.cc:374] T 00000000000000000000000000000000 P 211b80de3926480f9404708d45bfc84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:24:37.704172 25201 raft_consensus.cc:729] T 00000000000000000000000000000000 P 211b80de3926480f9404708d45bfc84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 211b80de3926480f9404708d45bfc84c, State: Initialized, Role: FOLLOWER
I20260925 15:24:37.704309 25201 consensus_queue.cc:260] T 00000000000000000000000000000000 P 211b80de3926480f9404708d45bfc84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211b80de3926480f9404708d45bfc84c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46535 } }
I20260925 15:24:37.704432 25201 raft_consensus.cc:388] T 00000000000000000000000000000000 P 211b80de3926480f9404708d45bfc84c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:24:37.704468 25201 raft_consensus.cc:482] T 00000000000000000000000000000000 P 211b80de3926480f9404708d45bfc84c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:24:37.704516 25201 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 211b80de3926480f9404708d45bfc84c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:37.705302 25201 raft_consensus.cc:504] T 00000000000000000000000000000000 P 211b80de3926480f9404708d45bfc84c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211b80de3926480f9404708d45bfc84c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46535 } }
I20260925 15:24:37.705415 25201 leader_election.cc:302] T 00000000000000000000000000000000 P 211b80de3926480f9404708d45bfc84c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 211b80de3926480f9404708d45bfc84c; no voters:
I20260925 15:24:37.705590 25201 leader_election.cc:288] T 00000000000000000000000000000000 P 211b80de3926480f9404708d45bfc84c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:24:37.705617 25206 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 211b80de3926480f9404708d45bfc84c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:24:37.705725 25206 raft_consensus.cc:686] T 00000000000000000000000000000000 P 211b80de3926480f9404708d45bfc84c [term 1 LEADER]: Becoming Leader. State: Replica: 211b80de3926480f9404708d45bfc84c, State: Running, Role: LEADER
I20260925 15:24:37.705819 25206 consensus_queue.cc:237] T 00000000000000000000000000000000 P 211b80de3926480f9404708d45bfc84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211b80de3926480f9404708d45bfc84c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46535 } }
I20260925 15:24:37.705888 25201 sys_catalog.cc:564] T 00000000000000000000000000000000 P 211b80de3926480f9404708d45bfc84c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:24:37.706152 25206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 211b80de3926480f9404708d45bfc84c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "211b80de3926480f9404708d45bfc84c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211b80de3926480f9404708d45bfc84c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46535 } } }
I20260925 15:24:37.706224 25206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 211b80de3926480f9404708d45bfc84c [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:37.706362 25206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 211b80de3926480f9404708d45bfc84c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 211b80de3926480f9404708d45bfc84c. Latest consensus state: current_term: 1 leader_uuid: "211b80de3926480f9404708d45bfc84c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "211b80de3926480f9404708d45bfc84c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46535 } } }
I20260925 15:24:37.706501 25206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 211b80de3926480f9404708d45bfc84c [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:37.706676 25210 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:24:37.707261 25210 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:24:37.709291 25210 catalog_manager.cc:1347] Generated new cluster ID: a47da2fca57746229834c75a17f9b7a3
I20260925 15:24:37.709350 25210 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 15:24:37 dist-test-slave-4bk3 krb5kdc[25120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 15:24:37.715999 25204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:37.701363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.79:43777 (local address 127.0.0.1:46535)
0925 15:24:37.701583 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:37.701597 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:37.701623 (+ 26us) server_negotiation.cc:407] Connection header received
0925 15:24:37.702312 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:37.702321 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:37.702404 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:37.702570 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:37.703511 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:37.705332 (+ 1821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:37.706573 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:37.707217 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:37.707341 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:37.710928 (+ 3587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:37.710932 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:37.710932 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:37.710947 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:37.713419 (+ 2472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:37.714297 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:37.714301 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:37.714302 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:37.714435 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:37.714853 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:37.714854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:37.714854 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:37.715205 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:37.715395 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:37.715657 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 15:24:37.715776 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":119,"threads_started":1}
I20260925 15:24:37.717809 3919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:24:37.727423 25204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:37.719531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.79:60737 (local address 127.0.0.1:46535)
0925 15:24:37.719575 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:37.719577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:37.719628 (+ 51us) server_negotiation.cc:407] Connection header received
0925 15:24:37.719694 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:37.719694 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:37.719705 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:37.719783 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:37.720467 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:37.721272 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:37.722310 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:37.722698 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:37.722778 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:37.724273 (+ 1495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:37.724275 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:37.724275 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:37.724283 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:37.725950 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:37.726404 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:37.726406 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:37.726406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:37.726470 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:37.726697 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:37.726698 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:37.726698 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:37.726840 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:37.726951 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:37.727241 (+ 290us) server_negotiation.cc:299] Negotiation successful
0925 15:24:37.727313 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 15:24:37.729673 25210 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:24:37.730464 25210 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:24:37.754318 25210 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 211b80de3926480f9404708d45bfc84c: Generated new TSK 0
I20260925 15:24:37.754557 25210 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:24:37.762883 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 25133
W20260925 15:24:37.767760 25118 connection.cc:535] client connection to 127.0.0.1:46535 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46535): Connection reset by peer (error 104)
2026-09-25T15:24:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1441 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349610595785-3919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:24:37 dist-test-slave-4bk3 krb5kdc[25232](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:24:37 dist-test-slave-4bk3 krb5kdc[25232](info): set up 2 sockets
Sep 25 15:24:37 dist-test-slave-4bk3 krb5kdc[25232](info): commencing operation
krb5kdc: starting...
W20260925 15:24:38.832700 3919 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Sep 25 15:24:38 dist-test-slave-4bk3 krb5kdc[25232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:24:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:24:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349610595785-3919-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349610595785-3919-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:24:38.969009 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37273
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349610595785-3919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37273
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:24:39.055806 25245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:39.056111 25245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:39.056157 25245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:39.057508 25245 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:24:39.057601 25245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:39.057626 25245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:39.057648 25245 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:24:39.057678 25245 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:24:39.059378 25245 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349610595785-3919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37273
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349610595785-3919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:24:39.059707 25245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:39.059978 25245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:39.065393 25250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:39.065719 25245 server_base.cc:1034] running on GCE node
W20260925 15:24:39.065395 25253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:39.065394 25251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:39.066085 25245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:39.066394 25245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:39.067548 25245 hybrid_clock.cc:648] HybridClock initialized: now 1790349879067503 us; error 70 us; skew 500 ppm
Sep 25 15:24:39 dist-test-slave-4bk3 krb5kdc[25232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349879, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:24:39.070509 25245 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:24:39.071050 25245 webserver.cc:460] Webserver started at http://127.0.0.1:34971/ using document root <none> and password file <none>
I20260925 15:24:39.071329 25245 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:39.071388 25245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:39.071522 25245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:39.072765 25245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349610595785-3919-0/minicluster-data/master-0/data/instance:
uuid: "910f5b918657467d867d943399468368"
format_stamp: "Formatted at 2026-09-25 15:24:39 on dist-test-slave-4bk3"
server_key: "10aae7739d8621e356008c7db5fb7651"
server_key_iv: "c46c366fa2c6209cb7649f20dc532b6b"
server_key_version: "encryptionkey@0"
I20260925 15:24:39.073144 25245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349610595785-3919-0/minicluster-data/master-0/wal/instance:
uuid: "910f5b918657467d867d943399468368"
format_stamp: "Formatted at 2026-09-25 15:24:39 on dist-test-slave-4bk3"
server_key: "10aae7739d8621e356008c7db5fb7651"
server_key_iv: "c46c366fa2c6209cb7649f20dc532b6b"
server_key_version: "encryptionkey@0"
I20260925 15:24:39.074625 25245 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:24:39.075610 25260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:39.075922 25245 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:24:39.076074 25245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349610595785-3919-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349610595785-3919-0/minicluster-data/master-0/wal
uuid: "910f5b918657467d867d943399468368"
format_stamp: "Formatted at 2026-09-25 15:24:39 on dist-test-slave-4bk3"
server_key: "10aae7739d8621e356008c7db5fb7651"
server_key_iv: "c46c366fa2c6209cb7649f20dc532b6b"
server_key_version: "encryptionkey@0"
I20260925 15:24:39.076167 25245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349610595785-3919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349610595785-3919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349610595785-3919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:39.090164 25245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:39.093182 25245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:39.093394 25245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:39.097743 25312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37273 every 8 connection(s)
I20260925 15:24:39.097755 25245 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37273
I20260925 15:24:39.098280 25245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
I20260925 15:24:39.099557 25313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:39.102980 25313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 910f5b918657467d867d943399468368: Bootstrap starting.
I20260925 15:24:39.103621 25313 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 910f5b918657467d867d943399468368: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:39.103880 25313 log.cc:824] T 00000000000000000000000000000000 P 910f5b918657467d867d943399468368: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:39.105111 25313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 910f5b918657467d867d943399468368: No bootstrap required, opened a new log
I20260925 15:24:39.106029 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 25245
I20260925 15:24:39.106248 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790349610595785-3919-0/minicluster-data/master-0/wal/instance
I20260925 15:24:39.106478 3919 external_mini_cluster.cc:1428] Setting key 3a80cd59b7ac0bc97c2aa6579fd15c7b
I20260925 15:24:39.107846 25313 raft_consensus.cc:348] T 00000000000000000000000000000000 P 910f5b918657467d867d943399468368 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "910f5b918657467d867d943399468368" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37273 } }
I20260925 15:24:39.108147 25313 raft_consensus.cc:374] T 00000000000000000000000000000000 P 910f5b918657467d867d943399468368 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:24:39.108197 25313 raft_consensus.cc:729] T 00000000000000000000000000000000 P 910f5b918657467d867d943399468368 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 910f5b918657467d867d943399468368, State: Initialized, Role: FOLLOWER
I20260925 15:24:39.108449 25313 consensus_queue.cc:260] T 00000000000000000000000000000000 P 910f5b918657467d867d943399468368 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "910f5b918657467d867d943399468368" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37273 } }
I20260925 15:24:39.108580 25313 raft_consensus.cc:388] T 00000000000000000000000000000000 P 910f5b918657467d867d943399468368 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:24:39.108620 25313 raft_consensus.cc:482] T 00000000000000000000000000000000 P 910f5b918657467d867d943399468368 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:24:39.108664 25313 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 910f5b918657467d867d943399468368 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:39.109977 25313 raft_consensus.cc:504] T 00000000000000000000000000000000 P 910f5b918657467d867d943399468368 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "910f5b918657467d867d943399468368" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37273 } }
I20260925 15:24:39.110185 25313 leader_election.cc:302] T 00000000000000000000000000000000 P 910f5b918657467d867d943399468368 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 910f5b918657467d867d943399468368; no voters:
I20260925 15:24:39.110387 25313 leader_election.cc:288] T 00000000000000000000000000000000 P 910f5b918657467d867d943399468368 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:24:39.110482 25318 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 910f5b918657467d867d943399468368 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:24:39.110671 25313 sys_catalog.cc:564] T 00000000000000000000000000000000 P 910f5b918657467d867d943399468368 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:24:39.110709 25318 raft_consensus.cc:686] T 00000000000000000000000000000000 P 910f5b918657467d867d943399468368 [term 1 LEADER]: Becoming Leader. State: Replica: 910f5b918657467d867d943399468368, State: Running, Role: LEADER
I20260925 15:24:39.110966 25318 consensus_queue.cc:237] T 00000000000000000000000000000000 P 910f5b918657467d867d943399468368 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "910f5b918657467d867d943399468368" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37273 } }
I20260925 15:24:39.111440 25320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 910f5b918657467d867d943399468368 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 910f5b918657467d867d943399468368. Latest consensus state: current_term: 1 leader_uuid: "910f5b918657467d867d943399468368" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "910f5b918657467d867d943399468368" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37273 } } }
I20260925 15:24:39.111548 25320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 910f5b918657467d867d943399468368 [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:39.111788 25319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 910f5b918657467d867d943399468368 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "910f5b918657467d867d943399468368" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "910f5b918657467d867d943399468368" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37273 } } }
I20260925 15:24:39.111907 25319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 910f5b918657467d867d943399468368 [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:39.111788 25327 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:24:39.112571 25327 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:24:39.114399 25327 catalog_manager.cc:1347] Generated new cluster ID: bcb1950a24574907bdfd3a25076735a7
I20260925 15:24:39.114483 25327 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:24:39.116983 25317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:39.107291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.79:43621 (local address 127.0.0.1:37273)
0925 15:24:39.107619 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:39.107635 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:39.107693 (+ 58us) server_negotiation.cc:407] Connection header received
0925 15:24:39.107865 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:39.107875 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:39.107955 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:39.108161 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:39.108613 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:39.110704 (+ 2091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:39.112145 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:39.112853 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:39.113346 (+ 493us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:39.113858 (+ 512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:39.113863 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:39.113863 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:24:39.114085 (+ 222us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:39.114117 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:24:39.114129 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:39.114250 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:39.116489 (+ 2239us) server_negotiation.cc:299] Negotiation successful
0925 15:24:39.116590 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":141,"threads_started":1}
I20260925 15:24:39.118634 3919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:24:39.124725 25317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:39.120360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.79:44377 (local address 127.0.0.1:37273)
0925 15:24:39.120403 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:39.120405 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:39.120571 (+ 166us) server_negotiation.cc:407] Connection header received
0925 15:24:39.120643 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:39.120644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:39.120656 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:39.120715 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:39.120884 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:39.121622 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:39.123006 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:39.123363 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:39.123454 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:39.124435 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:39.124437 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:39.124438 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:24:39.124444 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:39.124462 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:24:39.124467 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:39.124536 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:39.124620 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 15:24:39.124636 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 15:24:39.127220 25327 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:24:39.127813 25327 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:24:39.132423 25327 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 910f5b918657467d867d943399468368: Generated new TSK 0
I20260925 15:24:39.132638 25327 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:24:39.137462 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 25245
W20260925 15:24:39.142045 25230 connection.cc:535] client connection to 127.0.0.1:37273 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37273): Connection reset by peer (error 104)
2026-09-25T15:24:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1373 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349610595785-3919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:24:39 dist-test-slave-4bk3 krb5kdc[25344](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:24:39 dist-test-slave-4bk3 krb5kdc[25344](info): set up 2 sockets
Sep 25 15:24:39 dist-test-slave-4bk3 krb5kdc[25344](info): commencing operation
krb5kdc: starting...
W20260925 15:24:40.224596 3919 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Sep 25 15:24:40 dist-test-slave-4bk3 krb5kdc[25344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:24:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:24:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349610595785-3919-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349610595785-3919-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:24:40.356498 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40297
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349610595785-3919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40297
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:24:40.442342 25357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:40.442631 25357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:40.442673 25357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:40.444005 25357 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:24:40.444109 25357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:40.444133 25357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:40.444155 25357 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:24:40.444177 25357 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:24:40.446530 25357 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349610595785-3919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40297
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349610595785-3919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:24:40.446909 25357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:40.447176 25357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:40.452415 25362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:40.452442 25365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:40.452425 25363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:40.452728 25357 server_base.cc:1034] running on GCE node
I20260925 15:24:40.453011 25357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:40.453351 25357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:40.454537 25357 hybrid_clock.cc:648] HybridClock initialized: now 1790349880454492 us; error 68 us; skew 500 ppm
Sep 25 15:24:40 dist-test-slave-4bk3 krb5kdc[25344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349880, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:24:40.457521 25357 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:24:40.458038 25357 webserver.cc:460] Webserver started at http://127.0.0.1:39591/ using document root <none> and password file <none>
I20260925 15:24:40.458295 25357 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:40.458369 25357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:40.458523 25357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:40.459645 25357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349610595785-3919-0/minicluster-data/master-0/data/instance:
uuid: "bd88f6b56d17476e804ef4e3872cbfbc"
format_stamp: "Formatted at 2026-09-25 15:24:40 on dist-test-slave-4bk3"
server_key: "91ae1b45911b2fc05954694d69adf390"
server_key_iv: "ecdba5cb2296513b78e90e2e91d5f04c"
server_key_version: "encryptionkey@0"
I20260925 15:24:40.460031 25357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349610595785-3919-0/minicluster-data/master-0/wal/instance:
uuid: "bd88f6b56d17476e804ef4e3872cbfbc"
format_stamp: "Formatted at 2026-09-25 15:24:40 on dist-test-slave-4bk3"
server_key: "91ae1b45911b2fc05954694d69adf390"
server_key_iv: "ecdba5cb2296513b78e90e2e91d5f04c"
server_key_version: "encryptionkey@0"
I20260925 15:24:40.461615 25357 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:24:40.462507 25372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:40.462749 25357 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:24:40.462875 25357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349610595785-3919-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349610595785-3919-0/minicluster-data/master-0/wal
uuid: "bd88f6b56d17476e804ef4e3872cbfbc"
format_stamp: "Formatted at 2026-09-25 15:24:40 on dist-test-slave-4bk3"
server_key: "91ae1b45911b2fc05954694d69adf390"
server_key_iv: "ecdba5cb2296513b78e90e2e91d5f04c"
server_key_version: "encryptionkey@0"
I20260925 15:24:40.462958 25357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349610595785-3919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349610595785-3919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349610595785-3919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:40.466749 25357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:40.469898 25357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:40.470108 25357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:40.475872 25357 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40297
I20260925 15:24:40.475988 25424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40297 every 8 connection(s)
I20260925 15:24:40.476573 25357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
I20260925 15:24:40.477809 25425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:40.480417 25425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd88f6b56d17476e804ef4e3872cbfbc: Bootstrap starting.
I20260925 15:24:40.481053 25425 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd88f6b56d17476e804ef4e3872cbfbc: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:40.481328 25425 log.cc:824] T 00000000000000000000000000000000 P bd88f6b56d17476e804ef4e3872cbfbc: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:40.482259 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 25357
I20260925 15:24:40.482405 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790349610595785-3919-0/minicluster-data/master-0/wal/instance
I20260925 15:24:40.482549 25425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd88f6b56d17476e804ef4e3872cbfbc: No bootstrap required, opened a new log
I20260925 15:24:40.482632 3919 external_mini_cluster.cc:1428] Setting key bb84316fbb3105ea737e43674387d9ba
I20260925 15:24:40.484838 25425 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd88f6b56d17476e804ef4e3872cbfbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd88f6b56d17476e804ef4e3872cbfbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40297 } }
I20260925 15:24:40.485024 25425 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd88f6b56d17476e804ef4e3872cbfbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:24:40.485051 25425 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd88f6b56d17476e804ef4e3872cbfbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd88f6b56d17476e804ef4e3872cbfbc, State: Initialized, Role: FOLLOWER
I20260925 15:24:40.485209 25425 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd88f6b56d17476e804ef4e3872cbfbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd88f6b56d17476e804ef4e3872cbfbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40297 } }
I20260925 15:24:40.485343 25425 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd88f6b56d17476e804ef4e3872cbfbc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:24:40.485386 25425 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd88f6b56d17476e804ef4e3872cbfbc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:24:40.485623 25425 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd88f6b56d17476e804ef4e3872cbfbc [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:40.486886 25429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:40.483261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.79:55059 (local address 127.0.0.1:40297)
0925 15:24:40.483463 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:40.483471 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:40.483499 (+ 28us) server_negotiation.cc:407] Connection header received
0925 15:24:40.484870 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:40.484886 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:40.485016 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:40.485382 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:40.486362 (+ 980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:40.486366 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:40.486367 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:24:40.486372 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:40.486414 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:24:40.486427 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:40.486499 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:40.486597 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 15:24:40.486673 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":106,"threads_started":1}
I20260925 15:24:40.487039 25425 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd88f6b56d17476e804ef4e3872cbfbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd88f6b56d17476e804ef4e3872cbfbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40297 } }
I20260925 15:24:40.487177 25425 leader_election.cc:302] T 00000000000000000000000000000000 P bd88f6b56d17476e804ef4e3872cbfbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd88f6b56d17476e804ef4e3872cbfbc; no voters:
I20260925 15:24:40.487440 25425 leader_election.cc:288] T 00000000000000000000000000000000 P bd88f6b56d17476e804ef4e3872cbfbc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:24:40.487512 25430 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd88f6b56d17476e804ef4e3872cbfbc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:24:40.487711 25430 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd88f6b56d17476e804ef4e3872cbfbc [term 1 LEADER]: Becoming Leader. State: Replica: bd88f6b56d17476e804ef4e3872cbfbc, State: Running, Role: LEADER
I20260925 15:24:40.487775 25425 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd88f6b56d17476e804ef4e3872cbfbc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:24:40.487864 25430 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd88f6b56d17476e804ef4e3872cbfbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd88f6b56d17476e804ef4e3872cbfbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40297 } }
I20260925 15:24:40.488047 3919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:24:40.488317 25432 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd88f6b56d17476e804ef4e3872cbfbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd88f6b56d17476e804ef4e3872cbfbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd88f6b56d17476e804ef4e3872cbfbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40297 } } }
I20260925 15:24:40.488444 25432 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd88f6b56d17476e804ef4e3872cbfbc [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:40.488306 25430 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd88f6b56d17476e804ef4e3872cbfbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd88f6b56d17476e804ef4e3872cbfbc. Latest consensus state: current_term: 1 leader_uuid: "bd88f6b56d17476e804ef4e3872cbfbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd88f6b56d17476e804ef4e3872cbfbc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40297 } } }
I20260925 15:24:40.488632 25430 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd88f6b56d17476e804ef4e3872cbfbc [sys.catalog]: This master's current role is: LEADER
W20260925 15:24:40.490437 25448 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bd88f6b56d17476e804ef4e3872cbfbc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:24:40.490523 25448 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:24:40.490614 25438 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:24:40.490828 25438 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:24:40.491317 25429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:40.489869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.79:54281 (local address 127.0.0.1:40297)
0925 15:24:40.489912 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:40.489914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:40.490113 (+ 199us) server_negotiation.cc:407] Connection header received
0925 15:24:40.490223 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:40.490224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:40.490232 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:40.490263 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:40.491083 (+ 820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:40.491084 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:40.491085 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:24:40.491090 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:40.491106 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:24:40.491110 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:40.491170 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:40.491232 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 15:24:40.491248 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 15:24:40.493666 25438 catalog_manager.cc:1347] Generated new cluster ID: 5475c284ab4f4d8eb82f0d62c2de9040
I20260925 15:24:40.493752 25438 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:24:40.520994 25438 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:24:40.521337 25438 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:24:40.534188 25438 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd88f6b56d17476e804ef4e3872cbfbc: Generated new TSK 0
I20260925 15:24:40.534452 25438 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:24:40.538371 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 25357
2026-09-25T15:24:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1400 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349610595785-3919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:24:40 dist-test-slave-4bk3 krb5kdc[25456](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:24:40 dist-test-slave-4bk3 krb5kdc[25456](info): set up 2 sockets
Sep 25 15:24:40 dist-test-slave-4bk3 krb5kdc[25456](info): commencing operation
krb5kdc: starting...
W20260925 15:24:41.621155 3919 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Sep 25 15:24:41 dist-test-slave-4bk3 krb5kdc[25456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:24:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:24:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.211.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
I20260925 15:24:41.755057 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:32927
--webserver_interface=127.3.211.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:37585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.211.254:32927
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:24:41.846475 25469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:41.846765 25469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:41.846805 25469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:41.848054 25469 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:24:41.848146 25469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:41.848169 25469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:41.848189 25469 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:24:41.848217 25469 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:24:41.849886 25469 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:37585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349610595785-3919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.211.254:32927
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:32927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.211.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:24:41.850209 25469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:41.850468 25469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:41.856269 25475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:41.856272 25474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:41.856436 25469 server_base.cc:1034] running on GCE node
W20260925 15:24:41.856272 25477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:41.856765 25469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:41.857115 25469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:41.858279 25469 hybrid_clock.cc:648] HybridClock initialized: now 1790349881858245 us; error 47 us; skew 500 ppm
Sep 25 15:24:41 dist-test-slave-4bk3 krb5kdc[25456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349881, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:24:41.866078 25469 init.cc:377] Logged in from keytab as kudu/127.3.211.254@KRBTEST.COM (short username kudu)
I20260925 15:24:41.866575 25469 webserver.cc:460] Webserver started at http://127.3.211.254:42171/ using document root <none> and password file <none>
I20260925 15:24:41.866845 25469 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:41.866894 25469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:41.867058 25469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:41.868222 25469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349610595785-3919-0/minicluster-data/master-0/data/instance:
uuid: "fd998d594b9d400b90e8a8dc6fbbf3e0"
format_stamp: "Formatted at 2026-09-25 15:24:41 on dist-test-slave-4bk3"
server_key: "8cecef644695b68818b2ae29f5a2ff59"
server_key_iv: "57a8b4da13d8022477d6d16409ab6fe7"
server_key_version: "encryptionkey@0"
I20260925 15:24:41.868680 25469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349610595785-3919-0/minicluster-data/master-0/wal/instance:
uuid: "fd998d594b9d400b90e8a8dc6fbbf3e0"
format_stamp: "Formatted at 2026-09-25 15:24:41 on dist-test-slave-4bk3"
server_key: "8cecef644695b68818b2ae29f5a2ff59"
server_key_iv: "57a8b4da13d8022477d6d16409ab6fe7"
server_key_version: "encryptionkey@0"
I20260925 15:24:41.870147 25469 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:24:41.870998 25484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:41.871196 25469 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:24:41.871316 25469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349610595785-3919-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349610595785-3919-0/minicluster-data/master-0/wal
uuid: "fd998d594b9d400b90e8a8dc6fbbf3e0"
format_stamp: "Formatted at 2026-09-25 15:24:41 on dist-test-slave-4bk3"
server_key: "8cecef644695b68818b2ae29f5a2ff59"
server_key_iv: "57a8b4da13d8022477d6d16409ab6fe7"
server_key_version: "encryptionkey@0"
I20260925 15:24:41.871394 25469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349610595785-3919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349610595785-3919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349610595785-3919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:41.896726 25469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:41.899631 25469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:41.899770 25469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:41.905431 25469 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.254:32927
I20260925 15:24:41.905463 25536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.254:32927 every 8 connection(s)
I20260925 15:24:41.905845 25469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
I20260925 15:24:41.906738 25537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:41.909117 25537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd998d594b9d400b90e8a8dc6fbbf3e0: Bootstrap starting.
I20260925 15:24:41.909760 25537 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd998d594b9d400b90e8a8dc6fbbf3e0: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:41.910072 25537 log.cc:824] T 00000000000000000000000000000000 P fd998d594b9d400b90e8a8dc6fbbf3e0: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:41.910904 25537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd998d594b9d400b90e8a8dc6fbbf3e0: No bootstrap required, opened a new log
I20260925 15:24:41.911196 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 25469
I20260925 15:24:41.911302 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790349610595785-3919-0/minicluster-data/master-0/wal/instance
I20260925 15:24:41.911469 3919 external_mini_cluster.cc:1428] Setting key a6c6c54e6cbf9ca232988403df88d573
I20260925 15:24:41.913439 25537 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd998d594b9d400b90e8a8dc6fbbf3e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd998d594b9d400b90e8a8dc6fbbf3e0" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 32927 } }
I20260925 15:24:41.913640 25537 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd998d594b9d400b90e8a8dc6fbbf3e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:24:41.913679 25537 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd998d594b9d400b90e8a8dc6fbbf3e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd998d594b9d400b90e8a8dc6fbbf3e0, State: Initialized, Role: FOLLOWER
I20260925 15:24:41.913837 25537 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd998d594b9d400b90e8a8dc6fbbf3e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd998d594b9d400b90e8a8dc6fbbf3e0" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 32927 } }
I20260925 15:24:41.913913 25537 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd998d594b9d400b90e8a8dc6fbbf3e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:24:41.913935 25537 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd998d594b9d400b90e8a8dc6fbbf3e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:24:41.914007 25537 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd998d594b9d400b90e8a8dc6fbbf3e0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:41.915247 25537 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd998d594b9d400b90e8a8dc6fbbf3e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd998d594b9d400b90e8a8dc6fbbf3e0" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 32927 } }
I20260925 15:24:41.915371 25537 leader_election.cc:302] T 00000000000000000000000000000000 P fd998d594b9d400b90e8a8dc6fbbf3e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd998d594b9d400b90e8a8dc6fbbf3e0; no voters:
I20260925 15:24:41.915536 25537 leader_election.cc:288] T 00000000000000000000000000000000 P fd998d594b9d400b90e8a8dc6fbbf3e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:24:41.915673 25542 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd998d594b9d400b90e8a8dc6fbbf3e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:24:41.915900 25542 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd998d594b9d400b90e8a8dc6fbbf3e0 [term 1 LEADER]: Becoming Leader. State: Replica: fd998d594b9d400b90e8a8dc6fbbf3e0, State: Running, Role: LEADER
I20260925 15:24:41.916047 25542 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd998d594b9d400b90e8a8dc6fbbf3e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd998d594b9d400b90e8a8dc6fbbf3e0" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 32927 } }
I20260925 15:24:41.916105 25537 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd998d594b9d400b90e8a8dc6fbbf3e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:24:41.917156 25544 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd998d594b9d400b90e8a8dc6fbbf3e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd998d594b9d400b90e8a8dc6fbbf3e0. Latest consensus state: current_term: 1 leader_uuid: "fd998d594b9d400b90e8a8dc6fbbf3e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd998d594b9d400b90e8a8dc6fbbf3e0" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 32927 } } }
I20260925 15:24:41.917307 25544 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd998d594b9d400b90e8a8dc6fbbf3e0 [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:41.917505 25552 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:24:41.917749 25543 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd998d594b9d400b90e8a8dc6fbbf3e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd998d594b9d400b90e8a8dc6fbbf3e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd998d594b9d400b90e8a8dc6fbbf3e0" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 32927 } } }
I20260925 15:24:41.917927 25543 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd998d594b9d400b90e8a8dc6fbbf3e0 [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:41.918121 25552 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:24:41.919800 25552 catalog_manager.cc:1347] Generated new cluster ID: 075dcbfd031943a6ac59706ec66ebce0
I20260925 15:24:41.919859 25552 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 15:24:41 dist-test-slave-4bk3 krb5kdc[25456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:24:41.925973 25541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:41.912033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60599 (local address 127.3.211.254:32927)
0925 15:24:41.912255 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:41.912265 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:41.912300 (+ 35us) server_negotiation.cc:407] Connection header received
0925 15:24:41.913396 (+ 1096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:41.913406 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:41.913479 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:41.913647 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:41.914503 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:41.916105 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:41.918218 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:41.918920 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:41.919041 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:41.921825 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:41.921845 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:41.921845 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:41.921858 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:41.923628 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:41.924086 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:41.924091 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:41.924092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:41.924778 (+ 686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:41.925055 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:41.925056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:41.925057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:41.925347 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:41.925581 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:41.925674 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 15:24:41.925796 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":115,"threads_started":1}
I20260925 15:24:41.927057 3919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:24:41.935202 25541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:41.928668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44473 (local address 127.3.211.254:32927)
0925 15:24:41.928706 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:41.928708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:41.928718 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:24:41.928746 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:41.928747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:41.928757 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:41.928813 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:41.929375 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:41.929898 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:41.930624 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:41.930875 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:41.930975 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:41.932258 (+ 1283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:41.932259 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:41.932259 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:41.932267 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:41.933819 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:41.934223 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:41.934225 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:41.934225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:41.934290 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:41.934544 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:41.934545 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:41.934546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:41.934670 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:41.934801 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:41.934945 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 15:24:41.935013 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 15:24:41.975430 25552 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:24:41.976171 25552 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:24:41.986706 25552 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd998d594b9d400b90e8a8dc6fbbf3e0: Generated new TSK 0
I20260925 15:24:41.986999 25552 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:24:41.994716 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 25469
W20260925 15:24:41.999418 25454 connection.cc:535] client connection to 127.3.211.254:32927 recv error: Network error: failed to read from TLS socket (remote: 127.3.211.254:32927): Connection reset by peer (error 104)
2026-09-25T15:24:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1456 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349610595785-3919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:24:42 dist-test-slave-4bk3 krb5kdc[25568](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:24:42 dist-test-slave-4bk3 krb5kdc[25568](info): set up 2 sockets
Sep 25 15:24:42 dist-test-slave-4bk3 krb5kdc[25568](info): commencing operation
krb5kdc: starting...
W20260925 15:24:43.060616 3919 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Sep 25 15:24:43 dist-test-slave-4bk3 krb5kdc[25568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:24:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:24:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.211.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
I20260925 15:24:43.187892 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:44843
--webserver_interface=127.3.211.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:38791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.211.254:44843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:24:43.273842 25581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:43.274084 25581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:43.274111 25581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:43.276005 25581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:24:43.276098 25581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:43.276124 25581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:43.276155 25581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:24:43.276183 25581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:24:43.279170 25581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:38791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349610595785-3919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.211.254:44843
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:44843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.211.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:24:43.279484 25581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:43.279677 25581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:43.284996 25589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:43.284996 25587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:43.285053 25586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:43.285282 25581 server_base.cc:1034] running on GCE node
I20260925 15:24:43.285645 25581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:43.287717 25581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:43.288901 25581 hybrid_clock.cc:648] HybridClock initialized: now 1790349883288870 us; error 69 us; skew 500 ppm
Sep 25 15:24:43 dist-test-slave-4bk3 krb5kdc[25568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349883, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:24:43.291613 25581 init.cc:377] Logged in from keytab as kudu/127.3.211.254@KRBTEST.COM (short username kudu)
I20260925 15:24:43.292100 25581 webserver.cc:460] Webserver started at http://127.3.211.254:42409/ using document root <none> and password file <none>
I20260925 15:24:43.292398 25581 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:43.292485 25581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:43.292649 25581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:43.293794 25581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349610595785-3919-0/minicluster-data/master-0/data/instance:
uuid: "ad2ef575f2bf4bcb80f593f04278b2f2"
format_stamp: "Formatted at 2026-09-25 15:24:43 on dist-test-slave-4bk3"
server_key: "60ba8d3606ca4fdb2c556f2c9304f149"
server_key_iv: "3bbb45d396771886347168b9931b7976"
server_key_version: "encryptionkey@0"
I20260925 15:24:43.294185 25581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349610595785-3919-0/minicluster-data/master-0/wal/instance:
uuid: "ad2ef575f2bf4bcb80f593f04278b2f2"
format_stamp: "Formatted at 2026-09-25 15:24:43 on dist-test-slave-4bk3"
server_key: "60ba8d3606ca4fdb2c556f2c9304f149"
server_key_iv: "3bbb45d396771886347168b9931b7976"
server_key_version: "encryptionkey@0"
I20260925 15:24:43.295753 25581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:24:43.296936 25596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:43.297314 25581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:24:43.297433 25581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349610595785-3919-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349610595785-3919-0/minicluster-data/master-0/wal
uuid: "ad2ef575f2bf4bcb80f593f04278b2f2"
format_stamp: "Formatted at 2026-09-25 15:24:43 on dist-test-slave-4bk3"
server_key: "60ba8d3606ca4fdb2c556f2c9304f149"
server_key_iv: "3bbb45d396771886347168b9931b7976"
server_key_version: "encryptionkey@0"
I20260925 15:24:43.297534 25581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349610595785-3919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349610595785-3919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349610595785-3919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:43.326957 25581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:43.330082 25581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:43.330266 25581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:43.335662 25581 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.254:44843
I20260925 15:24:43.335703 25648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.254:44843 every 8 connection(s)
I20260925 15:24:43.336084 25581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
I20260925 15:24:43.337107 25649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:43.339706 25649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad2ef575f2bf4bcb80f593f04278b2f2: Bootstrap starting.
I20260925 15:24:43.340401 25649 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad2ef575f2bf4bcb80f593f04278b2f2: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:43.340695 25649 log.cc:824] T 00000000000000000000000000000000 P ad2ef575f2bf4bcb80f593f04278b2f2: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:43.341504 25649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad2ef575f2bf4bcb80f593f04278b2f2: No bootstrap required, opened a new log
I20260925 15:24:43.343227 25649 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad2ef575f2bf4bcb80f593f04278b2f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad2ef575f2bf4bcb80f593f04278b2f2" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 44843 } }
I20260925 15:24:43.343386 25649 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad2ef575f2bf4bcb80f593f04278b2f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:24:43.343410 25649 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad2ef575f2bf4bcb80f593f04278b2f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad2ef575f2bf4bcb80f593f04278b2f2, State: Initialized, Role: FOLLOWER
I20260925 15:24:43.343581 25649 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad2ef575f2bf4bcb80f593f04278b2f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad2ef575f2bf4bcb80f593f04278b2f2" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 44843 } }
I20260925 15:24:43.343644 25649 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad2ef575f2bf4bcb80f593f04278b2f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:24:43.343665 25649 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad2ef575f2bf4bcb80f593f04278b2f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:24:43.343732 25649 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad2ef575f2bf4bcb80f593f04278b2f2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:43.343945 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 25581
I20260925 15:24:43.344074 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790349610595785-3919-0/minicluster-data/master-0/wal/instance
I20260925 15:24:43.344271 3919 external_mini_cluster.cc:1428] Setting key 4a90a71c2ce065f1067f4506b92edb63
I20260925 15:24:43.344971 25649 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad2ef575f2bf4bcb80f593f04278b2f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad2ef575f2bf4bcb80f593f04278b2f2" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 44843 } }
I20260925 15:24:43.345099 25649 leader_election.cc:302] T 00000000000000000000000000000000 P ad2ef575f2bf4bcb80f593f04278b2f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad2ef575f2bf4bcb80f593f04278b2f2; no voters:
I20260925 15:24:43.345273 25649 leader_election.cc:288] T 00000000000000000000000000000000 P ad2ef575f2bf4bcb80f593f04278b2f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:24:43.345297 25654 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad2ef575f2bf4bcb80f593f04278b2f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:24:43.345419 25654 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad2ef575f2bf4bcb80f593f04278b2f2 [term 1 LEADER]: Becoming Leader. State: Replica: ad2ef575f2bf4bcb80f593f04278b2f2, State: Running, Role: LEADER
I20260925 15:24:43.345541 25649 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad2ef575f2bf4bcb80f593f04278b2f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:24:43.345561 25654 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad2ef575f2bf4bcb80f593f04278b2f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad2ef575f2bf4bcb80f593f04278b2f2" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 44843 } }
I20260925 15:24:43.345993 25655 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad2ef575f2bf4bcb80f593f04278b2f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad2ef575f2bf4bcb80f593f04278b2f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad2ef575f2bf4bcb80f593f04278b2f2" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 44843 } } }
I20260925 15:24:43.346010 25656 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad2ef575f2bf4bcb80f593f04278b2f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad2ef575f2bf4bcb80f593f04278b2f2. Latest consensus state: current_term: 1 leader_uuid: "ad2ef575f2bf4bcb80f593f04278b2f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad2ef575f2bf4bcb80f593f04278b2f2" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 44843 } } }
I20260925 15:24:43.346104 25655 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad2ef575f2bf4bcb80f593f04278b2f2 [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:43.346117 25656 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad2ef575f2bf4bcb80f593f04278b2f2 [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:43.346352 25661 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:24:43.346971 25661 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:24:43.349568 25661 catalog_manager.cc:1347] Generated new cluster ID: a33ea135cf5b4b21b866a035117f7a6e
I20260925 15:24:43.349655 25661 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:24:43.350615 25653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:43.344825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51025 (local address 127.3.211.254:44843)
0925 15:24:43.345042 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:43.345070 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:43.345112 (+ 42us) server_negotiation.cc:407] Connection header received
0925 15:24:43.345265 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:43.345270 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:43.345348 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:43.345731 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:43.345839 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:43.347376 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:43.348448 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:43.348880 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:43.349023 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:43.349853 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:43.349857 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:43.349857 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:24:43.350067 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:43.350103 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:24:43.350118 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:43.350187 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:43.350364 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 15:24:43.350394 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":124,"threads_started":1}
I20260925 15:24:43.352113 3919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:24:43.358011 25653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:43.353944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48013 (local address 127.3.211.254:44843)
0925 15:24:43.353991 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:43.353993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:43.354004 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:24:43.354057 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:43.354058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:43.354070 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:43.354139 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:43.354280 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:43.355459 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:43.356507 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:43.356846 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:43.356928 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:43.357732 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:43.357734 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:43.357734 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:24:43.357739 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:43.357756 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:24:43.357760 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:43.357832 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:43.357907 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 15:24:43.357922 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 15:24:43.374001 25661 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:24:43.374596 25661 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:24:43.380041 25661 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad2ef575f2bf4bcb80f593f04278b2f2: Generated new TSK 0
I20260925 15:24:43.380304 25661 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:24:43.390895 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 25581
W20260925 15:24:43.395723 25566 connection.cc:535] client connection to 127.3.211.254:44843 recv error: Network error: failed to read from TLS socket (remote: 127.3.211.254:44843): Connection reset by peer (error 104)
2026-09-25T15:24:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1396 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349610595785-3919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:24:43 dist-test-slave-4bk3 krb5kdc[25680](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:24:43 dist-test-slave-4bk3 krb5kdc[25680](info): set up 2 sockets
Sep 25 15:24:43 dist-test-slave-4bk3 krb5kdc[25680](info): commencing operation
krb5kdc: starting...
W20260925 15:24:44.456700 3919 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Sep 25 15:24:44 dist-test-slave-4bk3 krb5kdc[25680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:24:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:24:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.211.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
I20260925 15:24:44.587334 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:42761
--webserver_interface=127.3.211.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:42139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.211.254:42761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:24:44.677273 25693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:44.677523 25693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:44.677551 25693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:44.678835 25693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:24:44.678898 25693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:44.678913 25693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:44.678926 25693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:24:44.678938 25693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:24:44.680640 25693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:42139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349610595785-3919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.211.254:42761
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:42761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.211.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:24:44.680945 25693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:44.681195 25693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:44.686370 25699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:44.686395 25701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:44.686501 25698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:44.687217 25693 server_base.cc:1034] running on GCE node
I20260925 15:24:44.687542 25693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:44.687885 25693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:44.689060 25693 hybrid_clock.cc:648] HybridClock initialized: now 1790349884689036 us; error 35 us; skew 500 ppm
Sep 25 15:24:44 dist-test-slave-4bk3 krb5kdc[25680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349884, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:24:44.692266 25693 init.cc:377] Logged in from keytab as kudu/127.3.211.254@KRBTEST.COM (short username kudu)
I20260925 15:24:44.693017 25693 webserver.cc:460] Webserver started at http://127.3.211.254:36939/ using document root <none> and password file <none>
I20260925 15:24:44.693346 25693 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:44.693406 25693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:44.693563 25693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:44.694984 25693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349610595785-3919-0/minicluster-data/master-0/data/instance:
uuid: "3c661fef27b842c2a604e9ad6bfc87c0"
format_stamp: "Formatted at 2026-09-25 15:24:44 on dist-test-slave-4bk3"
server_key: "29d91e492442a793c5f086a44dace292"
server_key_iv: "8e1625b1b442df2c36edf1d76deb40fc"
server_key_version: "encryptionkey@0"
I20260925 15:24:44.695434 25693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349610595785-3919-0/minicluster-data/master-0/wal/instance:
uuid: "3c661fef27b842c2a604e9ad6bfc87c0"
format_stamp: "Formatted at 2026-09-25 15:24:44 on dist-test-slave-4bk3"
server_key: "29d91e492442a793c5f086a44dace292"
server_key_iv: "8e1625b1b442df2c36edf1d76deb40fc"
server_key_version: "encryptionkey@0"
I20260925 15:24:44.697525 25693 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260925 15:24:44.698719 25708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:44.699110 25693 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:24:44.699281 25693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349610595785-3919-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349610595785-3919-0/minicluster-data/master-0/wal
uuid: "3c661fef27b842c2a604e9ad6bfc87c0"
format_stamp: "Formatted at 2026-09-25 15:24:44 on dist-test-slave-4bk3"
server_key: "29d91e492442a793c5f086a44dace292"
server_key_iv: "8e1625b1b442df2c36edf1d76deb40fc"
server_key_version: "encryptionkey@0"
I20260925 15:24:44.699397 25693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349610595785-3919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349610595785-3919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349610595785-3919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:44.703114 25693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:44.705789 25693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:44.706007 25693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:44.711568 25693 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.254:42761
I20260925 15:24:44.711609 25760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.254:42761 every 8 connection(s)
I20260925 15:24:44.712026 25693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
I20260925 15:24:44.712899 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 25693
I20260925 15:24:44.713032 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790349610595785-3919-0/minicluster-data/master-0/wal/instance
I20260925 15:24:44.713238 3919 external_mini_cluster.cc:1428] Setting key 03f334630e688db9efdaac8e6786c8b8
I20260925 15:24:44.713280 25761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:44.717018 25764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:44.714066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42629 (local address 127.3.211.254:42761)
0925 15:24:44.714328 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:44.714338 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:44.714370 (+ 32us) server_negotiation.cc:407] Connection header received
0925 15:24:44.715163 (+ 793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:44.715175 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:44.715215 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:44.715402 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:44.716397 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:44.716402 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:44.716403 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:24:44.716412 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:44.716469 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:24:44.716482 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:44.716559 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:44.716787 (+ 228us) server_negotiation.cc:299] Negotiation successful
0925 15:24:44.716843 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":120,"threads_started":1}
I20260925 15:24:44.717487 25761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c661fef27b842c2a604e9ad6bfc87c0: Bootstrap starting.
I20260925 15:24:44.718294 25761 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c661fef27b842c2a604e9ad6bfc87c0: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:44.718396 3919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:24:44.718603 25761 log.cc:824] T 00000000000000000000000000000000 P 3c661fef27b842c2a604e9ad6bfc87c0: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:44.719348 25761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c661fef27b842c2a604e9ad6bfc87c0: No bootstrap required, opened a new log
I20260925 15:24:44.722879 25764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:44.720759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40563 (local address 127.3.211.254:42761)
0925 15:24:44.720818 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:44.720821 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:44.721173 (+ 352us) server_negotiation.cc:407] Connection header received
0925 15:24:44.721276 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:44.721277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:44.721282 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:44.721371 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:44.722360 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:44.722361 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:44.722361 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:24:44.722366 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:44.722382 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:24:44.722386 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:44.722454 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:44.722571 (+ 117us) server_negotiation.cc:299] Negotiation successful
0925 15:24:44.722585 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 15:24:44.722748 25761 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c661fef27b842c2a604e9ad6bfc87c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c661fef27b842c2a604e9ad6bfc87c0" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 42761 } }
I20260925 15:24:44.723136 25761 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c661fef27b842c2a604e9ad6bfc87c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:24:44.723240 25761 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c661fef27b842c2a604e9ad6bfc87c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c661fef27b842c2a604e9ad6bfc87c0, State: Initialized, Role: FOLLOWER
I20260925 15:24:44.723402 25761 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c661fef27b842c2a604e9ad6bfc87c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c661fef27b842c2a604e9ad6bfc87c0" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 42761 } }
I20260925 15:24:44.723557 25761 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c661fef27b842c2a604e9ad6bfc87c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:24:44.723609 25761 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c661fef27b842c2a604e9ad6bfc87c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:24:44.723667 25761 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c661fef27b842c2a604e9ad6bfc87c0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:44.724438 25761 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c661fef27b842c2a604e9ad6bfc87c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c661fef27b842c2a604e9ad6bfc87c0" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 42761 } }
I20260925 15:24:44.724546 25761 leader_election.cc:302] T 00000000000000000000000000000000 P 3c661fef27b842c2a604e9ad6bfc87c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c661fef27b842c2a604e9ad6bfc87c0; no voters:
I20260925 15:24:44.724750 25761 leader_election.cc:288] T 00000000000000000000000000000000 P 3c661fef27b842c2a604e9ad6bfc87c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:24:44.724848 25771 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c661fef27b842c2a604e9ad6bfc87c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:24:44.725016 25761 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c661fef27b842c2a604e9ad6bfc87c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:24:44.725039 25771 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c661fef27b842c2a604e9ad6bfc87c0 [term 1 LEADER]: Becoming Leader. State: Replica: 3c661fef27b842c2a604e9ad6bfc87c0, State: Running, Role: LEADER
I20260925 15:24:44.725138 25771 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c661fef27b842c2a604e9ad6bfc87c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c661fef27b842c2a604e9ad6bfc87c0" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 42761 } }
I20260925 15:24:44.725518 25772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c661fef27b842c2a604e9ad6bfc87c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c661fef27b842c2a604e9ad6bfc87c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c661fef27b842c2a604e9ad6bfc87c0" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 42761 } } }
I20260925 15:24:44.725523 25773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c661fef27b842c2a604e9ad6bfc87c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c661fef27b842c2a604e9ad6bfc87c0. Latest consensus state: current_term: 1 leader_uuid: "3c661fef27b842c2a604e9ad6bfc87c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c661fef27b842c2a604e9ad6bfc87c0" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 42761 } } }
I20260925 15:24:44.725668 25772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c661fef27b842c2a604e9ad6bfc87c0 [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:44.725732 25773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c661fef27b842c2a604e9ad6bfc87c0 [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:44.726588 25780 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:24:44.727317 25780 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:24:44.729509 25780 catalog_manager.cc:1347] Generated new cluster ID: 42c6fb64b7f047b9b835b5a86f2a6c58
I20260925 15:24:44.729588 25780 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:24:44.741788 25780 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:24:44.741990 25780 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:24:44.750613 25780 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c661fef27b842c2a604e9ad6bfc87c0: Generated new TSK 0
I20260925 15:24:44.750870 25780 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:24:44.762302 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 25693
2026-09-25T15:24:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1371 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349610595785-3919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:24:44 dist-test-slave-4bk3 krb5kdc[25792](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:24:44 dist-test-slave-4bk3 krb5kdc[25792](info): set up 2 sockets
Sep 25 15:24:44 dist-test-slave-4bk3 krb5kdc[25792](info): commencing operation
krb5kdc: starting...
W20260925 15:24:45.828846 3919 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Sep 25 15:24:45 dist-test-slave-4bk3 krb5kdc[25792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:24:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:24:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.211.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
I20260925 15:24:45.961117 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:35711
--webserver_interface=127.3.211.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:46529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.211.254:35711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:24:46.051252 25805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:46.051491 25805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:46.051517 25805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:46.052783 25805 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:24:46.052838 25805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:46.052852 25805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:46.052865 25805 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:24:46.052878 25805 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:24:46.054467 25805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:46529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349610595785-3919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.211.254:35711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:35711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.211.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:24:46.054749 25805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:46.054989 25805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:46.060009 25813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:46.060014 25810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:46.060212 25811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:46.060549 25805 server_base.cc:1034] running on GCE node
I20260925 15:24:46.060868 25805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:46.061206 25805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:46.062402 25805 hybrid_clock.cc:648] HybridClock initialized: now 1790349886062376 us; error 45 us; skew 500 ppm
Sep 25 15:24:46 dist-test-slave-4bk3 krb5kdc[25792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349886, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:24:46.065297 25805 init.cc:377] Logged in from keytab as kudu/127.3.211.254@KRBTEST.COM (short username kudu)
I20260925 15:24:46.065866 25805 webserver.cc:460] Webserver started at http://127.3.211.254:38609/ using document root <none> and password file <none>
I20260925 15:24:46.066118 25805 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:46.066190 25805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:46.066352 25805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:46.067664 25805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349610595785-3919-0/minicluster-data/master-0/data/instance:
uuid: "7d1a4fbb0fa945b2ac431833346f8e8c"
format_stamp: "Formatted at 2026-09-25 15:24:46 on dist-test-slave-4bk3"
server_key: "6703f6e05d1d519c13575eefad7a3a6e"
server_key_iv: "23a6c19d48afe84655e8cc32749a18a0"
server_key_version: "encryptionkey@0"
I20260925 15:24:46.068111 25805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349610595785-3919-0/minicluster-data/master-0/wal/instance:
uuid: "7d1a4fbb0fa945b2ac431833346f8e8c"
format_stamp: "Formatted at 2026-09-25 15:24:46 on dist-test-slave-4bk3"
server_key: "6703f6e05d1d519c13575eefad7a3a6e"
server_key_iv: "23a6c19d48afe84655e8cc32749a18a0"
server_key_version: "encryptionkey@0"
I20260925 15:24:46.069681 25805 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 15:24:46.070766 25820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:46.071142 25805 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:24:46.071297 25805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349610595785-3919-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349610595785-3919-0/minicluster-data/master-0/wal
uuid: "7d1a4fbb0fa945b2ac431833346f8e8c"
format_stamp: "Formatted at 2026-09-25 15:24:46 on dist-test-slave-4bk3"
server_key: "6703f6e05d1d519c13575eefad7a3a6e"
server_key_iv: "23a6c19d48afe84655e8cc32749a18a0"
server_key_version: "encryptionkey@0"
I20260925 15:24:46.071388 25805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349610595785-3919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349610595785-3919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349610595785-3919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:46.093071 25805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:46.095693 25805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:46.095883 25805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:46.100282 25805 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.254:35711
I20260925 15:24:46.100440 25872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.254:35711 every 8 connection(s)
I20260925 15:24:46.100840 25805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
I20260925 15:24:46.101872 25873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:46.104404 25873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d1a4fbb0fa945b2ac431833346f8e8c: Bootstrap starting.
I20260925 15:24:46.105034 25873 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d1a4fbb0fa945b2ac431833346f8e8c: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:46.105345 25873 log.cc:824] T 00000000000000000000000000000000 P 7d1a4fbb0fa945b2ac431833346f8e8c: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:46.106178 25873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d1a4fbb0fa945b2ac431833346f8e8c: No bootstrap required, opened a new log
I20260925 15:24:46.107728 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 25805
I20260925 15:24:46.107851 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790349610595785-3919-0/minicluster-data/master-0/wal/instance
I20260925 15:24:46.108044 3919 external_mini_cluster.cc:1428] Setting key 4d29dcca77377bb6397d74c587501044
I20260925 15:24:46.108121 25873 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d1a4fbb0fa945b2ac431833346f8e8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d1a4fbb0fa945b2ac431833346f8e8c" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 35711 } }
I20260925 15:24:46.108299 25873 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d1a4fbb0fa945b2ac431833346f8e8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:24:46.108371 25873 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d1a4fbb0fa945b2ac431833346f8e8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d1a4fbb0fa945b2ac431833346f8e8c, State: Initialized, Role: FOLLOWER
I20260925 15:24:46.108560 25873 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d1a4fbb0fa945b2ac431833346f8e8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d1a4fbb0fa945b2ac431833346f8e8c" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 35711 } }
I20260925 15:24:46.108649 25873 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d1a4fbb0fa945b2ac431833346f8e8c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:24:46.108690 25873 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d1a4fbb0fa945b2ac431833346f8e8c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:24:46.108750 25873 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d1a4fbb0fa945b2ac431833346f8e8c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:46.109887 25873 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d1a4fbb0fa945b2ac431833346f8e8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d1a4fbb0fa945b2ac431833346f8e8c" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 35711 } }
I20260925 15:24:46.109994 25873 leader_election.cc:302] T 00000000000000000000000000000000 P 7d1a4fbb0fa945b2ac431833346f8e8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d1a4fbb0fa945b2ac431833346f8e8c; no voters:
I20260925 15:24:46.110183 25873 leader_election.cc:288] T 00000000000000000000000000000000 P 7d1a4fbb0fa945b2ac431833346f8e8c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:24:46.110282 25878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d1a4fbb0fa945b2ac431833346f8e8c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:24:46.110455 25878 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d1a4fbb0fa945b2ac431833346f8e8c [term 1 LEADER]: Becoming Leader. State: Replica: 7d1a4fbb0fa945b2ac431833346f8e8c, State: Running, Role: LEADER
I20260925 15:24:46.110527 25873 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d1a4fbb0fa945b2ac431833346f8e8c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:24:46.110572 25878 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d1a4fbb0fa945b2ac431833346f8e8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d1a4fbb0fa945b2ac431833346f8e8c" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 35711 } }
I20260925 15:24:46.110910 25878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d1a4fbb0fa945b2ac431833346f8e8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d1a4fbb0fa945b2ac431833346f8e8c. Latest consensus state: current_term: 1 leader_uuid: "7d1a4fbb0fa945b2ac431833346f8e8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d1a4fbb0fa945b2ac431833346f8e8c" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 35711 } } }
I20260925 15:24:46.110918 25880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d1a4fbb0fa945b2ac431833346f8e8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d1a4fbb0fa945b2ac431833346f8e8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d1a4fbb0fa945b2ac431833346f8e8c" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 35711 } } }
I20260925 15:24:46.111013 25880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d1a4fbb0fa945b2ac431833346f8e8c [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:46.111012 25878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d1a4fbb0fa945b2ac431833346f8e8c [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:46.111222 25885 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:24:46.111764 25885 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:24:46.113482 25885 catalog_manager.cc:1347] Generated new cluster ID: 0b6434deb4ff464b8d31d1e4e8033e2d
I20260925 15:24:46.113539 25885 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 15:24:46 dist-test-slave-4bk3 krb5kdc[25792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:24:46.120606 25877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:46.108616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40053 (local address 127.3.211.254:35711)
0925 15:24:46.108845 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:46.108855 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:46.109045 (+ 190us) server_negotiation.cc:407] Connection header received
0925 15:24:46.109193 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:46.109201 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:46.109273 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:46.109486 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:46.110174 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:46.111712 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:46.112918 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:46.113542 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:46.113683 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:46.116637 (+ 2954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:46.116641 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:46.116641 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:46.116654 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:46.118645 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:46.119108 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:46.119111 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:46.119112 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:46.119232 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:46.119467 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:46.119467 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:46.119468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:46.119792 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:46.119949 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:46.120221 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 15:24:46.120394 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":100,"threads_started":1}
I20260925 15:24:46.121925 3919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:24:46.123370 25885 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:24:46.124055 25885 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:24:46.129902 25877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:46.123481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37169 (local address 127.3.211.254:35711)
0925 15:24:46.123519 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:46.123521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:46.123647 (+ 126us) server_negotiation.cc:407] Connection header received
0925 15:24:46.123704 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:46.123705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:46.123715 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:46.123771 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:46.124319 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:46.124863 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:46.125569 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:46.125790 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:46.125869 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:46.127135 (+ 1266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:46.127136 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:46.127137 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:46.127145 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:46.128657 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:46.129058 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:46.129060 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:46.129060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:46.129121 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:46.129340 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:46.129340 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:46.129341 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:46.129458 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:46.129581 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:46.129699 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 15:24:46.129794 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 15:24:46.133912 25885 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d1a4fbb0fa945b2ac431833346f8e8c: Generated new TSK 0
I20260925 15:24:46.134138 25885 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:24:46.138566 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 25805
2026-09-25T15:24:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1375 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349610595785-3919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:24:46 dist-test-slave-4bk3 krb5kdc[25904](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:24:46 dist-test-slave-4bk3 krb5kdc[25904](info): set up 2 sockets
Sep 25 15:24:46 dist-test-slave-4bk3 krb5kdc[25904](info): commencing operation
krb5kdc: starting...
W20260925 15:24:47.197569 3919 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Sep 25 15:24:47 dist-test-slave-4bk3 krb5kdc[25904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:24:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:24:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.211.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
I20260925 15:24:47.328910 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:40907
--webserver_interface=127.3.211.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:33781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.211.254:40907
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:24:47.437667 25917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:47.437945 25917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:47.437992 25917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:47.439402 25917 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:24:47.439499 25917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:47.439523 25917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:47.439558 25917 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:24:47.439585 25917 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:24:47.441480 25917 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:33781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349610595785-3919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.211.254:40907
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:40907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.211.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:24:47.441802 25917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:47.442045 25917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:47.447359 25923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:47.447359 25922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:47.447692 25925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:47.448016 25917 server_base.cc:1034] running on GCE node
I20260925 15:24:47.448392 25917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:47.448763 25917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:47.449952 25917 hybrid_clock.cc:648] HybridClock initialized: now 1790349887449934 us; error 47 us; skew 500 ppm
Sep 25 15:24:47 dist-test-slave-4bk3 krb5kdc[25904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349887, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:24:47.452723 25917 init.cc:377] Logged in from keytab as kudu/127.3.211.254@KRBTEST.COM (short username kudu)
I20260925 15:24:47.453258 25917 webserver.cc:460] Webserver started at http://127.3.211.254:44659/ using document root <none> and password file <none>
I20260925 15:24:47.453517 25917 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:47.453565 25917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:47.453698 25917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:47.454818 25917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349610595785-3919-0/minicluster-data/master-0/data/instance:
uuid: "5a5d234d453c4001bcd70698ed989024"
format_stamp: "Formatted at 2026-09-25 15:24:47 on dist-test-slave-4bk3"
server_key: "c0d26f56456a6ce5b718cf30cc92b504"
server_key_iv: "474a48271684cba06c399c80529d7638"
server_key_version: "encryptionkey@0"
I20260925 15:24:47.455216 25917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349610595785-3919-0/minicluster-data/master-0/wal/instance:
uuid: "5a5d234d453c4001bcd70698ed989024"
format_stamp: "Formatted at 2026-09-25 15:24:47 on dist-test-slave-4bk3"
server_key: "c0d26f56456a6ce5b718cf30cc92b504"
server_key_iv: "474a48271684cba06c399c80529d7638"
server_key_version: "encryptionkey@0"
I20260925 15:24:47.456812 25917 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:24:47.457816 25932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:47.458050 25917 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:24:47.458181 25917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349610595785-3919-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349610595785-3919-0/minicluster-data/master-0/wal
uuid: "5a5d234d453c4001bcd70698ed989024"
format_stamp: "Formatted at 2026-09-25 15:24:47 on dist-test-slave-4bk3"
server_key: "c0d26f56456a6ce5b718cf30cc92b504"
server_key_iv: "474a48271684cba06c399c80529d7638"
server_key_version: "encryptionkey@0"
I20260925 15:24:47.458274 25917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349610595785-3919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349610595785-3919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349610595785-3919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:47.483990 25917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:47.487365 25917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:47.487628 25917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:47.492565 25917 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.254:40907
I20260925 15:24:47.492674 25984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.254:40907 every 8 connection(s)
I20260925 15:24:47.493111 25917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
I20260925 15:24:47.494154 25985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:47.496435 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 25917
I20260925 15:24:47.496562 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790349610595785-3919-0/minicluster-data/master-0/wal/instance
I20260925 15:24:47.496752 3919 external_mini_cluster.cc:1428] Setting key eaf8457c6f4046cf9d32e51ae6b89f2e
I20260925 15:24:47.497644 25985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a5d234d453c4001bcd70698ed989024: Bootstrap starting.
I20260925 15:24:47.498395 25985 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a5d234d453c4001bcd70698ed989024: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:47.498749 25985 log.cc:824] T 00000000000000000000000000000000 P 5a5d234d453c4001bcd70698ed989024: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:47.499603 25985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a5d234d453c4001bcd70698ed989024: No bootstrap required, opened a new log
I20260925 15:24:47.502051 25985 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a5d234d453c4001bcd70698ed989024 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a5d234d453c4001bcd70698ed989024" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 40907 } }
I20260925 15:24:47.502270 25985 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a5d234d453c4001bcd70698ed989024 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:24:47.502319 25985 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a5d234d453c4001bcd70698ed989024 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a5d234d453c4001bcd70698ed989024, State: Initialized, Role: FOLLOWER
I20260925 15:24:47.502486 25985 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a5d234d453c4001bcd70698ed989024 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a5d234d453c4001bcd70698ed989024" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 40907 } }
I20260925 15:24:47.502588 25985 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a5d234d453c4001bcd70698ed989024 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:24:47.502624 25985 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a5d234d453c4001bcd70698ed989024 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:24:47.502681 25985 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a5d234d453c4001bcd70698ed989024 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:47.503481 25985 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a5d234d453c4001bcd70698ed989024 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a5d234d453c4001bcd70698ed989024" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 40907 } }
I20260925 15:24:47.503631 25985 leader_election.cc:302] T 00000000000000000000000000000000 P 5a5d234d453c4001bcd70698ed989024 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a5d234d453c4001bcd70698ed989024; no voters:
I20260925 15:24:47.503834 25985 leader_election.cc:288] T 00000000000000000000000000000000 P 5a5d234d453c4001bcd70698ed989024 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:24:47.503885 25990 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a5d234d453c4001bcd70698ed989024 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:24:47.504091 25990 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a5d234d453c4001bcd70698ed989024 [term 1 LEADER]: Becoming Leader. State: Replica: 5a5d234d453c4001bcd70698ed989024, State: Running, Role: LEADER
I20260925 15:24:47.504146 25985 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a5d234d453c4001bcd70698ed989024 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:24:47.504261 25988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:47.497410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42009 (local address 127.3.211.254:40907)
0925 15:24:47.497654 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:47.497665 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:47.497699 (+ 34us) server_negotiation.cc:407] Connection header received
0925 15:24:47.498331 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:47.498340 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:47.498407 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:47.498832 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:47.498944 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:47.500289 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:47.501410 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:47.502227 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:47.502784 (+ 557us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:47.503215 (+ 431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:47.503219 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:47.503219 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:24:47.503428 (+ 209us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:47.503462 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:24:47.503474 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:47.503535 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:47.503668 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 15:24:47.503748 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":123,"threads_started":1}
I20260925 15:24:47.504280 25990 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a5d234d453c4001bcd70698ed989024 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a5d234d453c4001bcd70698ed989024" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 40907 } }
I20260925 15:24:47.505376 3919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:24:47.506872 25992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a5d234d453c4001bcd70698ed989024 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a5d234d453c4001bcd70698ed989024. Latest consensus state: current_term: 1 leader_uuid: "5a5d234d453c4001bcd70698ed989024" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a5d234d453c4001bcd70698ed989024" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 40907 } } }
I20260925 15:24:47.507059 25992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a5d234d453c4001bcd70698ed989024 [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:47.507145 25991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a5d234d453c4001bcd70698ed989024 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a5d234d453c4001bcd70698ed989024" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a5d234d453c4001bcd70698ed989024" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 40907 } } }
I20260925 15:24:47.507457 25991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a5d234d453c4001bcd70698ed989024 [sys.catalog]: This master's current role is: LEADER
W20260925 15:24:47.507676 26009 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5a5d234d453c4001bcd70698ed989024: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:24:47.507711 26009 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:24:47.507776 26012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:24:47.507941 26012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:24:47.510072 26012 catalog_manager.cc:1347] Generated new cluster ID: 0493dbae7cd148cf9f1a0b942ebbfe95
I20260925 15:24:47.510143 26012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:24:47.510455 25988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:47.506854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43019 (local address 127.3.211.254:40907)
0925 15:24:47.506899 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:47.506901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:47.507066 (+ 165us) server_negotiation.cc:407] Connection header received
0925 15:24:47.507151 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:47.507152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:47.507167 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:47.507241 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:47.507427 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:47.507981 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:47.509041 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:47.509331 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:47.509421 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:47.510142 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:47.510143 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:47.510144 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:24:47.510150 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:47.510167 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:24:47.510172 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:47.510204 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:47.510333 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 15:24:47.510359 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 15:24:47.529879 26012 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:24:47.530730 26012 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:24:47.538269 26012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a5d234d453c4001bcd70698ed989024: Generated new TSK 0
I20260925 15:24:47.538535 26012 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:24:47.544720 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 25917
2026-09-25T15:24:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1408 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349610595785-3919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:24:47 dist-test-slave-4bk3 krb5kdc[26016](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:24:47 dist-test-slave-4bk3 krb5kdc[26016](info): set up 2 sockets
Sep 25 15:24:47 dist-test-slave-4bk3 krb5kdc[26016](info): commencing operation
krb5kdc: starting...
W20260925 15:24:48.608711 3919 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Sep 25 15:24:48 dist-test-slave-4bk3 krb5kdc[26016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:24:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:24:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.211.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
I20260925 15:24:48.740752 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:41213
--webserver_interface=127.3.211.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:45839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.211.254:41213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:24:48.826092 26029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:48.826393 26029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:48.826463 26029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:48.828073 26029 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:24:48.828162 26029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:48.828192 26029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:48.828223 26029 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:24:48.828274 26029 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:24:48.830185 26029 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:45839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349610595785-3919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.211.254:41213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:41213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.211.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:24:48.830534 26029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:48.830796 26029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:48.836197 26035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:48.836179 26034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:48.836499 26029 server_base.cc:1034] running on GCE node
W20260925 15:24:48.836179 26037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:48.836879 26029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:48.837206 26029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:48.838407 26029 hybrid_clock.cc:648] HybridClock initialized: now 1790349888838380 us; error 45 us; skew 500 ppm
Sep 25 15:24:48 dist-test-slave-4bk3 krb5kdc[26016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349888, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:24:48.840993 26029 init.cc:377] Logged in from keytab as kudu/127.3.211.254@KRBTEST.COM (short username kudu)
I20260925 15:24:48.841478 26029 webserver.cc:460] Webserver started at http://127.3.211.254:32927/ using document root <none> and password file <none>
I20260925 15:24:48.841686 26029 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:48.841732 26029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:48.841823 26029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:48.842876 26029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349610595785-3919-0/minicluster-data/master-0/data/instance:
uuid: "11b41ed4c42e4c51b57151b3a4025e26"
format_stamp: "Formatted at 2026-09-25 15:24:48 on dist-test-slave-4bk3"
server_key: "e1fd396d715ab5b4eb189f87c6e42909"
server_key_iv: "d98d6c5ed940de9581ddb239fb3159a1"
server_key_version: "encryptionkey@0"
I20260925 15:24:48.843884 26029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349610595785-3919-0/minicluster-data/master-0/wal/instance:
uuid: "11b41ed4c42e4c51b57151b3a4025e26"
format_stamp: "Formatted at 2026-09-25 15:24:48 on dist-test-slave-4bk3"
server_key: "e1fd396d715ab5b4eb189f87c6e42909"
server_key_iv: "d98d6c5ed940de9581ddb239fb3159a1"
server_key_version: "encryptionkey@0"
I20260925 15:24:48.845495 26029 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260925 15:24:48.846405 26044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:48.846627 26029 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:24:48.846721 26029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349610595785-3919-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349610595785-3919-0/minicluster-data/master-0/wal
uuid: "11b41ed4c42e4c51b57151b3a4025e26"
format_stamp: "Formatted at 2026-09-25 15:24:48 on dist-test-slave-4bk3"
server_key: "e1fd396d715ab5b4eb189f87c6e42909"
server_key_iv: "d98d6c5ed940de9581ddb239fb3159a1"
server_key_version: "encryptionkey@0"
I20260925 15:24:48.846781 26029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349610595785-3919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349610595785-3919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349610595785-3919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:48.850152 26029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:48.852624 26029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:48.852744 26029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:48.856614 26029 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.254:41213
I20260925 15:24:48.856652 26096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.254:41213 every 8 connection(s)
I20260925 15:24:48.857034 26029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
I20260925 15:24:48.858515 26097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:48.862406 26097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11b41ed4c42e4c51b57151b3a4025e26: Bootstrap starting.
I20260925 15:24:48.863328 26097 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11b41ed4c42e4c51b57151b3a4025e26: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:48.863688 26097 log.cc:824] T 00000000000000000000000000000000 P 11b41ed4c42e4c51b57151b3a4025e26: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:48.864739 26097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11b41ed4c42e4c51b57151b3a4025e26: No bootstrap required, opened a new log
I20260925 15:24:48.866169 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 26029
I20260925 15:24:48.866263 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790349610595785-3919-0/minicluster-data/master-0/wal/instance
I20260925 15:24:48.866472 3919 external_mini_cluster.cc:1428] Setting key cbd713475b709f9ec132b5adecce0323
I20260925 15:24:48.867033 26097 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11b41ed4c42e4c51b57151b3a4025e26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11b41ed4c42e4c51b57151b3a4025e26" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 41213 } }
I20260925 15:24:48.867251 26097 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11b41ed4c42e4c51b57151b3a4025e26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:24:48.867300 26097 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11b41ed4c42e4c51b57151b3a4025e26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11b41ed4c42e4c51b57151b3a4025e26, State: Initialized, Role: FOLLOWER
I20260925 15:24:48.867467 26097 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11b41ed4c42e4c51b57151b3a4025e26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11b41ed4c42e4c51b57151b3a4025e26" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 41213 } }
I20260925 15:24:48.867568 26097 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11b41ed4c42e4c51b57151b3a4025e26 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:24:48.867599 26097 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11b41ed4c42e4c51b57151b3a4025e26 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:24:48.867640 26097 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11b41ed4c42e4c51b57151b3a4025e26 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:48.868597 26097 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11b41ed4c42e4c51b57151b3a4025e26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11b41ed4c42e4c51b57151b3a4025e26" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 41213 } }
I20260925 15:24:48.868722 26097 leader_election.cc:302] T 00000000000000000000000000000000 P 11b41ed4c42e4c51b57151b3a4025e26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11b41ed4c42e4c51b57151b3a4025e26; no voters:
I20260925 15:24:48.868932 26097 leader_election.cc:288] T 00000000000000000000000000000000 P 11b41ed4c42e4c51b57151b3a4025e26 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:24:48.869060 26102 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11b41ed4c42e4c51b57151b3a4025e26 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:24:48.869159 26101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:48.867134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47351 (local address 127.3.211.254:41213)
0925 15:24:48.867452 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:48.867462 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:48.867499 (+ 37us) server_negotiation.cc:407] Connection header received
0925 15:24:48.867636 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:48.867643 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:48.867676 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:48.867833 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:48.868560 (+ 727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:48.868564 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:48.868564 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:24:48.868569 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:48.868613 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:24:48.868629 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:48.868707 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:48.868890 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 15:24:48.868941 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":105,"threads_started":1}
I20260925 15:24:48.869192 26102 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11b41ed4c42e4c51b57151b3a4025e26 [term 1 LEADER]: Becoming Leader. State: Replica: 11b41ed4c42e4c51b57151b3a4025e26, State: Running, Role: LEADER
I20260925 15:24:48.869277 26097 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11b41ed4c42e4c51b57151b3a4025e26 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:24:48.869270 26102 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11b41ed4c42e4c51b57151b3a4025e26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11b41ed4c42e4c51b57151b3a4025e26" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 41213 } }
I20260925 15:24:48.870342 3919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:24:48.870477 26104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11b41ed4c42e4c51b57151b3a4025e26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11b41ed4c42e4c51b57151b3a4025e26. Latest consensus state: current_term: 1 leader_uuid: "11b41ed4c42e4c51b57151b3a4025e26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11b41ed4c42e4c51b57151b3a4025e26" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 41213 } } }
I20260925 15:24:48.870584 26104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11b41ed4c42e4c51b57151b3a4025e26 [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:48.871028 26112 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:24:48.871538 26112 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:24:48.871775 26103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11b41ed4c42e4c51b57151b3a4025e26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11b41ed4c42e4c51b57151b3a4025e26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11b41ed4c42e4c51b57151b3a4025e26" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 41213 } } }
I20260925 15:24:48.871944 26103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11b41ed4c42e4c51b57151b3a4025e26 [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:48.873685 26101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:48.872147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42205 (local address 127.3.211.254:41213)
0925 15:24:48.872202 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:48.872204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:48.872369 (+ 165us) server_negotiation.cc:407] Connection header received
0925 15:24:48.872531 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:48.872533 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:48.872544 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:48.872629 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:48.873394 (+ 765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:48.873395 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:48.873396 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:24:48.873400 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:48.873417 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:24:48.873422 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:48.873501 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:48.873596 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 15:24:48.873612 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 15:24:48.873968 26112 catalog_manager.cc:1347] Generated new cluster ID: 33818bbfa76945a5a51cb4f5943e0185
I20260925 15:24:48.874048 26112 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:24:48.902925 26112 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:24:48.903134 26112 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:24:48.905344 26112 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11b41ed4c42e4c51b57151b3a4025e26: Generated new TSK 0
I20260925 15:24:48.905545 26112 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:24:48.916671 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 26029
2026-09-25T15:24:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1370 ms)
[----------] 18 tests from AuthTokenIssuingTest (25106 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:24:48 dist-test-slave-4bk3 krb5kdc[26128](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:24:48 dist-test-slave-4bk3 krb5kdc[26128](info): set up 2 sockets
Sep 25 15:24:48 dist-test-slave-4bk3 krb5kdc[26128](info): commencing operation
krb5kdc: starting...
W20260925 15:24:49.984659 3919 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Sep 25 15:24:50 dist-test-slave-4bk3 krb5kdc[26128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:24:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:24:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.211.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
I20260925 15:24:50.120517 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:46873
--webserver_interface=127.3.211.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:41547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.211.254:46873,127.3.211.253:42085,127.3.211.252:46311
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:24:50.206389 26141 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 15:24:50.206636 26141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:50.206656 26141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:50.206678 26141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:50.208016 26141 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:24:50.208063 26141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:50.208078 26141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:50.208091 26141 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:24:50.208103 26141 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:24:50.209730 26141 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:41547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.211.254:46873,127.3.211.253:42085,127.3.211.252:46311
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:46873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.211.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:24:50.209967 26141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:50.210150 26141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:50.215204 26147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:50.215175 26146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:50.215191 26149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:50.215418 26141 server_base.cc:1034] running on GCE node
I20260925 15:24:50.215755 26141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:50.216161 26141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:50.217336 26141 hybrid_clock.cc:648] HybridClock initialized: now 1790349890217318 us; error 33 us; skew 500 ppm
Sep 25 15:24:50 dist-test-slave-4bk3 krb5kdc[26128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349890, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:24:50.219915 26141 init.cc:377] Logged in from keytab as kudu/127.3.211.254@KRBTEST.COM (short username kudu)
I20260925 15:24:50.220513 26141 webserver.cc:460] Webserver started at http://127.3.211.254:35571/ using document root <none> and password file <none>
I20260925 15:24:50.220754 26141 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:50.220825 26141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:50.220981 26141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:50.222108 26141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/minicluster-data/master-0/data/instance:
uuid: "7f3f8b7fe1834d6299ee852a55932f15"
format_stamp: "Formatted at 2026-09-25 15:24:50 on dist-test-slave-4bk3"
server_key: "62eb132ae32453a565244ab049f7c2a4"
server_key_iv: "405598d3b7672a2aa18b6f3552694aa5"
server_key_version: "encryptionkey@0"
I20260925 15:24:50.222510 26141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/minicluster-data/master-0/wal/instance:
uuid: "7f3f8b7fe1834d6299ee852a55932f15"
format_stamp: "Formatted at 2026-09-25 15:24:50 on dist-test-slave-4bk3"
server_key: "62eb132ae32453a565244ab049f7c2a4"
server_key_iv: "405598d3b7672a2aa18b6f3552694aa5"
server_key_version: "encryptionkey@0"
I20260925 15:24:50.223963 26141 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:24:50.224869 26156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:50.225036 26141 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 15:24:50.225149 26141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/minicluster-data/master-0/wal
uuid: "7f3f8b7fe1834d6299ee852a55932f15"
format_stamp: "Formatted at 2026-09-25 15:24:50 on dist-test-slave-4bk3"
server_key: "62eb132ae32453a565244ab049f7c2a4"
server_key_iv: "405598d3b7672a2aa18b6f3552694aa5"
server_key_version: "encryptionkey@0"
I20260925 15:24:50.225231 26141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:50.242914 26141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:50.245834 26141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:50.246043 26141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:50.250242 26141 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.254:46873
I20260925 15:24:50.250275 26208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.254:46873 every 8 connection(s)
I20260925 15:24:50.250795 26141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
I20260925 15:24:50.252043 26209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:50.253700 26209 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.211.254" port: 46873 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:24:50.256731 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 26141
I20260925 15:24:50.256873 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/minicluster-data/master-0/wal/instance
I20260925 15:24:50.257085 3919 external_mini_cluster.cc:1428] Setting key 48c13900c90e798f4f0e609a63dde88e
Sep 25 15:24:50 dist-test-slave-4bk3 krb5kdc[26128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349890, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.254@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:24:50.265625 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:50.254370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:46873 (local address 127.0.0.1:56732)
0925 15:24:50.254601 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:50.254646 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:50.255122 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:50.255480 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:50.255483 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:50.255753 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:24:50.255938 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:50.255943 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:50.257240 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:50.257241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:50.258399 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:50.258402 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:50.259164 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:50.259166 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:50.259223 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:50.259867 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:24:50.259875 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:24:50.261657 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:24:50.263920 (+ 2263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:50.263929 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:50.263931 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:50.264145 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:50.264471 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:50.264472 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:50.264472 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:50.264574 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:50.265103 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:24:50.265110 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:24:50.265173 (+ 63us) client_negotiation.cc:769] Sending connection context
0925 15:24:50.265256 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 15:24:50.265404 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":143,"threads_started":1}
I20260925 15:24:50.265626 26212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:50.254453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56732 (local address 127.3.211.254:46873)
0925 15:24:50.255261 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:50.255264 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:50.255276 (+ 12us) server_negotiation.cc:407] Connection header received
0925 15:24:50.255335 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:50.255336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:50.255352 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:50.255431 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:50.256099 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:50.257126 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:50.258508 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:50.259048 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:50.259171 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:50.261763 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:50.261774 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:50.261774 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:50.261782 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:50.263803 (+ 2021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:50.264245 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:50.264249 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:50.264249 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:50.264387 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:50.264652 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:50.264653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:50.264654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:50.265014 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:50.265132 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:50.265327 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 15:24:50.265495 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":787,"thread_start_us":75,"threads_started":1}
WARNING: no policy specified for kudu/127.3.211.253@KRBTEST.COM; defaulting to no policy
I20260925 15:24:50.266563 26209 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42085 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:24:50.267015 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:50.266877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.253:42085 (local address 127.0.0.1:41880)
0925 15:24:50.266938 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:50.266989 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.253:42085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 15:24:50.267231 26159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.211.253:42085: connect: Connection refused (error 111)
I20260925 15:24:50.267800 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:50.267680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.253:42085 (local address 127.0.0.1:41896)
0925 15:24:50.267749 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:50.267778 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.253:42085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 15:24:50.268002 26209 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.211.253:42085: Network error: Client connection negotiation failed: client connection to 127.3.211.253:42085: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.253.keytab.
Entry for principal kudu/127.3.211.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.253.keytab.
I20260925 15:24:50.285691 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.253:42085
--webserver_interface=127.3.211.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.253.keytab
--principal=kudu/127.3.211.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:41547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.211.254:46873,127.3.211.253:42085,127.3.211.252:46311
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:24:50.317276 26209 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42085 } attempt: 1
I20260925 15:24:50.317797 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:50.317655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.253:42085 (local address 127.0.0.1:41906)
0925 15:24:50.317725 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:50.317766 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.253:42085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 15:24:50.318140 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:50.318039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.253:42085 (local address 127.0.0.1:41910)
0925 15:24:50.318088 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:50.318116 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.253:42085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 15:24:50.318226 26209 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.211.253:42085: Network error: Client connection negotiation failed: client connection to 127.3.211.253:42085: connect: Connection refused (error 111)
W20260925 15:24:50.375243 26216 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 15:24:50.375536 26216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:50.375566 26216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:50.375614 26216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:50.376889 26216 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:24:50.376981 26216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:50.377004 26216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:50.377027 26216 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:24:50.377055 26216 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:24:50.378703 26216 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:41547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.211.254:46873,127.3.211.253:42085,127.3.211.252:46311
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.253.keytab
--principal=kudu/127.3.211.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.253:42085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.211.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:24:50.379017 26216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:50.379280 26216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:50.384467 26221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:50.384483 26224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:50.384878 26216 server_base.cc:1034] running on GCE node
W20260925 15:24:50.384490 26222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:50.385241 26216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:50.385612 26216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:50.386384 26209 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42085 } attempt: 2
I20260925 15:24:50.386819 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:50.386673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.253:42085 (local address 127.0.0.1:41918)
0925 15:24:50.386744 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:50.386777 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.253:42085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 15:24:50.386808 26216 hybrid_clock.cc:648] HybridClock initialized: now 1790349890386784 us; error 47 us; skew 500 ppm
I20260925 15:24:50.387117 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:50.387046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.253:42085 (local address 127.0.0.1:41920)
0925 15:24:50.387073 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:50.387096 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.253:42085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260925 15:24:50.387220 26209 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.211.253:42085: Network error: Client connection negotiation failed: client connection to 127.3.211.253:42085: connect: Connection refused (error 111)
Sep 25 15:24:50 dist-test-slave-4bk3 krb5kdc[26128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349890, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:24:50.389686 26216 init.cc:377] Logged in from keytab as kudu/127.3.211.253@KRBTEST.COM (short username kudu)
I20260925 15:24:50.390257 26216 webserver.cc:460] Webserver started at http://127.3.211.253:39293/ using document root <none> and password file <none>
I20260925 15:24:50.390519 26216 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:50.390594 26216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:50.390748 26216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:50.391938 26216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/minicluster-data/master-1/data/instance:
uuid: "4b752c0b1bf849a29b6bfce9150f1bd3"
format_stamp: "Formatted at 2026-09-25 15:24:50 on dist-test-slave-4bk3"
server_key: "2fd83edbec6b6ac84933387ad01ce423"
server_key_iv: "2f9e4ce1a5751b032541c072e2b980da"
server_key_version: "encryptionkey@0"
I20260925 15:24:50.392401 26216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/minicluster-data/master-1/wal/instance:
uuid: "4b752c0b1bf849a29b6bfce9150f1bd3"
format_stamp: "Formatted at 2026-09-25 15:24:50 on dist-test-slave-4bk3"
server_key: "2fd83edbec6b6ac84933387ad01ce423"
server_key_iv: "2f9e4ce1a5751b032541c072e2b980da"
server_key_version: "encryptionkey@0"
I20260925 15:24:50.393978 26216 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:24:50.395135 26231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:50.395453 26216 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260925 15:24:50.395573 26216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/minicluster-data/master-1/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/minicluster-data/master-1/wal
uuid: "4b752c0b1bf849a29b6bfce9150f1bd3"
format_stamp: "Formatted at 2026-09-25 15:24:50 on dist-test-slave-4bk3"
server_key: "2fd83edbec6b6ac84933387ad01ce423"
server_key_iv: "2f9e4ce1a5751b032541c072e2b980da"
server_key_version: "encryptionkey@0"
I20260925 15:24:50.395670 26216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:50.418474 26216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:50.420902 26216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:50.421077 26216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:50.425212 26216 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.253:42085
I20260925 15:24:50.425251 26283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.253:42085 every 8 connection(s)
I20260925 15:24:50.425663 26216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/minicluster-data/master-1/data/info.pb
I20260925 15:24:50.426923 26284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:50.429072 26284 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.211.254" port: 46873 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:24:50.432366 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 26216
I20260925 15:24:50.432473 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/minicluster-data/master-1/wal/instance
I20260925 15:24:50.432680 3919 external_mini_cluster.cc:1428] Setting key 05f214f1c64140e263191250fa36ce09
Sep 25 15:24:50 dist-test-slave-4bk3 krb5kdc[26128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349890, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.253@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:24:50.439987 26212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:50.429999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56742 (local address 127.3.211.254:46873)
0925 15:24:50.430072 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:50.430075 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:50.430320 (+ 245us) server_negotiation.cc:407] Connection header received
0925 15:24:50.430830 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:50.430831 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:50.430846 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:50.430915 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:50.431576 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:50.432083 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:50.433710 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:50.433929 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:50.433991 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:50.436408 (+ 2417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:50.436411 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:50.436411 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:50.436420 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:50.438393 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:50.438887 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:50.438889 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:50.438889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:50.438964 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:50.439246 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:50.439247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:50.439248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:50.439421 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:50.439496 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:50.439767 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 15:24:50.439856 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 15:24:50.440081 26286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:50.429931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:46873 (local address 127.0.0.1:56742)
0925 15:24:50.430179 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:50.430220 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:50.430717 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:50.431005 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:50.431008 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:50.431279 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:24:50.431480 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:50.431483 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:50.432197 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:50.432198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:50.433607 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:50.433610 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:50.434040 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:50.434042 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:50.434225 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:50.434812 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:24:50.434846 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:24:50.436267 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:24:50.438514 (+ 2247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:50.438525 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:50.438527 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:50.438800 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:50.439041 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:50.439042 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:50.439042 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:50.439174 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:50.439493 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:24:50.439498 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:24:50.439628 (+ 130us) client_negotiation.cc:769] Sending connection context
0925 15:24:50.439715 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 15:24:50.439860 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":128,"threads_started":1}
I20260925 15:24:50.440737 26284 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42085 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.211.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.252@KRBTEST.COM" created.
Sep 25 15:24:50 dist-test-slave-4bk3 krb5kdc[26128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349890, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.253@KRBTEST.COM for kudu/127.3.211.253@KRBTEST.COM
I20260925 15:24:50.448743 26286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:50.441051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.253:42085 (local address 127.0.0.1:41924)
0925 15:24:50.441114 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:50.441123 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:50.441166 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:50.441447 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:50.441448 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:50.441591 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:24:50.441639 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:50.441640 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:50.442516 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:50.442517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:50.443012 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:50.443013 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:50.443582 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:50.443583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:50.443646 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:50.444202 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:24:50.444206 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:24:50.445589 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:24:50.447562 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:50.447563 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:50.447563 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:50.447753 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:50.447999 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:50.448000 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:50.448001 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:50.448056 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:50.448399 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:24:50.448400 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:24:50.448440 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 15:24:50.448513 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 15:24:50.448624 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 15:24:50.448807 26288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:50.441132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41924 (local address 127.3.211.253:42085)
0925 15:24:50.441312 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:50.441315 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:50.441332 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:24:50.441348 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:50.441350 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:50.441360 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:50.441446 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:50.441740 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:50.442416 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:50.443106 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:50.443478 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:50.443584 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:50.445693 (+ 2109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:50.445703 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:50.445703 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:50.445710 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:50.447425 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:50.447864 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:50.447867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:50.447868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:50.447928 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:50.448128 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:50.448129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:50.448130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:50.448294 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:50.448422 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:50.448544 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 15:24:50.448679 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":156,"thread_start_us":109,"threads_started":1}
I20260925 15:24:50.449909 26284 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.211.252" port: 46311 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:24:50.450338 26286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:50.450172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.252:46311 (local address 127.0.0.1:42966)
0925 15:24:50.450258 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:50.450314 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.252:46311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 15:24:50.450484 26232 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.211.252:46311: connect: Connection refused (error 111)
I20260925 15:24:50.450892 26286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:50.450797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.252:46311 (local address 127.0.0.1:42972)
0925 15:24:50.450843 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:50.450868 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.252:46311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 15:24:50.451015 26284 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.211.252:46311: Network error: Client connection negotiation failed: client connection to 127.3.211.252:46311: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.252.keytab.
Entry for principal kudu/127.3.211.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.252.keytab.
I20260925 15:24:50.461447 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.252:46311
--webserver_interface=127.3.211.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.252.keytab
--principal=kudu/127.3.211.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:41547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.211.254:46873,127.3.211.253:42085,127.3.211.252:46311
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:24:50.478391 26209 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42085 } attempt: 3
Sep 25 15:24:50 dist-test-slave-4bk3 krb5kdc[26128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349890, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.254@KRBTEST.COM for kudu/127.3.211.253@KRBTEST.COM
I20260925 15:24:50.485865 26288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:50.478917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41940 (local address 127.3.211.253:42085)
0925 15:24:50.478990 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:50.478997 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:50.479006 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:24:50.479032 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:50.479033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:50.479043 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:50.479108 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:50.479488 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:50.480011 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:50.481061 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:50.481271 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:50.481328 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:50.483385 (+ 2057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:50.483386 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:50.483387 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:50.483394 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:50.484779 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:50.485179 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:50.485180 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:50.485180 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:50.485230 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:50.485421 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:50.485422 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:50.485422 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:50.485527 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:50.485577 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:50.485736 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 15:24:50.485799 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 15:24:50.485911 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:50.478843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.253:42085 (local address 127.0.0.1:41940)
0925 15:24:50.478911 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:50.478922 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:50.478965 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:50.479134 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:50.479135 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:50.479335 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:24:50.479409 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:50.479411 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:50.480102 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:50.480104 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:50.480969 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:50.480972 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:50.481345 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:50.481346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:50.481421 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:50.481871 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:24:50.481876 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:24:50.483278 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:24:50.484867 (+ 1589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:50.484868 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:50.484868 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:50.485105 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:50.485296 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:50.485297 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:50.485297 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:50.485360 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:50.485588 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:24:50.485589 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:24:50.485640 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 15:24:50.485712 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 15:24:50.485810 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 15:24:50.486572 26209 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.211.252" port: 46311 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:24:50.487174 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:50.486904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.252:46311 (local address 127.0.0.1:42976)
0925 15:24:50.486941 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:50.487121 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.252:46311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260925 15:24:50.487584 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:50.487479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.252:46311 (local address 127.0.0.1:42992)
0925 15:24:50.487509 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:50.487561 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.252:46311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260925 15:24:50.487679 26209 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.211.252:46311: Network error: Client connection negotiation failed: client connection to 127.3.211.252:46311: connect: Connection refused (error 111)
I20260925 15:24:50.500226 26284 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.211.252" port: 46311 } attempt: 1
I20260925 15:24:50.500820 26286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:50.500676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.252:46311 (local address 127.0.0.1:42996)
0925 15:24:50.500742 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:50.500793 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.252:46311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 15:24:50.501250 26286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:50.501116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.252:46311 (local address 127.0.0.1:43000)
0925 15:24:50.501185 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:50.501226 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.252:46311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 15:24:50.501380 26284 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.211.252:46311: Network error: Client connection negotiation failed: client connection to 127.3.211.252:46311: connect: Connection refused (error 111)
I20260925 15:24:50.518838 26209 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.211.252" port: 46311 } attempt: 1
I20260925 15:24:50.519416 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:50.519270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.252:46311 (local address 127.0.0.1:43016)
0925 15:24:50.519339 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:50.519382 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.252:46311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 15:24:50.519812 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:50.519723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.252:46311 (local address 127.0.0.1:43030)
0925 15:24:50.519770 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:50.519796 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.252:46311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 15:24:50.519940 26209 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.211.252:46311: Network error: Client connection negotiation failed: client connection to 127.3.211.252:46311: connect: Connection refused (error 111)
W20260925 15:24:50.549350 26291 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 15:24:50.549592 26291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:50.549610 26291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:50.549633 26291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:50.550797 26291 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:24:50.550839 26291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:50.550853 26291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:50.550866 26291 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:24:50.550879 26291 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:24:50.552729 26291 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:41547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.211.254:46873,127.3.211.253:42085,127.3.211.252:46311
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.252.keytab
--principal=kudu/127.3.211.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.252:46311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.211.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:24:50.552989 26291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:50.553205 26291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:50.558456 26297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:50.558465 26299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:50.558511 26296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:50.558902 26291 server_base.cc:1034] running on GCE node
I20260925 15:24:50.559161 26291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:50.559469 26291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:50.560637 26291 hybrid_clock.cc:648] HybridClock initialized: now 1790349890560624 us; error 29 us; skew 500 ppm
Sep 25 15:24:50 dist-test-slave-4bk3 krb5kdc[26128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349890, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:24:50.563261 26291 init.cc:377] Logged in from keytab as kudu/127.3.211.252@KRBTEST.COM (short username kudu)
I20260925 15:24:50.563807 26291 webserver.cc:460] Webserver started at http://127.3.211.252:42995/ using document root <none> and password file <none>
I20260925 15:24:50.564061 26291 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:50.564137 26291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:50.564297 26291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:50.565469 26291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/minicluster-data/master-2/data/instance:
uuid: "a9a51cdae15b4a31aeae568604b23f75"
format_stamp: "Formatted at 2026-09-25 15:24:50 on dist-test-slave-4bk3"
server_key: "ca780fb6d00fbe2ed9830569f30c4b3b"
server_key_iv: "9a808612e7daf967f51c6933bf45b805"
server_key_version: "encryptionkey@0"
I20260925 15:24:50.565864 26291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/minicluster-data/master-2/wal/instance:
uuid: "a9a51cdae15b4a31aeae568604b23f75"
format_stamp: "Formatted at 2026-09-25 15:24:50 on dist-test-slave-4bk3"
server_key: "ca780fb6d00fbe2ed9830569f30c4b3b"
server_key_iv: "9a808612e7daf967f51c6933bf45b805"
server_key_version: "encryptionkey@0"
I20260925 15:24:50.567370 26291 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:24:50.568404 26306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:50.568610 26291 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:24:50.568732 26291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/minicluster-data/master-2/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/minicluster-data/master-2/wal
uuid: "a9a51cdae15b4a31aeae568604b23f75"
format_stamp: "Formatted at 2026-09-25 15:24:50 on dist-test-slave-4bk3"
server_key: "ca780fb6d00fbe2ed9830569f30c4b3b"
server_key_iv: "9a808612e7daf967f51c6933bf45b805"
server_key_version: "encryptionkey@0"
I20260925 15:24:50.568826 26291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:50.569604 26284 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.211.252" port: 46311 } attempt: 2
I20260925 15:24:50.570058 26286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:50.569923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.252:46311 (local address 127.0.0.1:43032)
0925 15:24:50.569988 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:50.570030 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.252:46311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 15:24:50.570384 26286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:50.570281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.252:46311 (local address 127.0.0.1:43038)
0925 15:24:50.570330 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:50.570355 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.252:46311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 15:24:50.570469 26284 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.211.252:46311: Network error: Client connection negotiation failed: client connection to 127.3.211.252:46311: connect: Connection refused (error 111)
I20260925 15:24:50.595156 26209 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.211.252" port: 46311 } attempt: 2
I20260925 15:24:50.595697 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:50.595539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.252:46311 (local address 127.0.0.1:43042)
0925 15:24:50.595617 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:50.595666 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.252:46311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 15:24:50.596233 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:50.596001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.252:46311 (local address 127.0.0.1:43052)
0925 15:24:50.596070 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:50.596118 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.252:46311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 15:24:50.596446 26209 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.211.252:46311: Network error: Client connection negotiation failed: client connection to 127.3.211.252:46311: connect: Connection refused (error 111)
I20260925 15:24:50.603319 26291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:50.605804 26291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:50.605969 26291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:50.610329 26291 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.252:46311
I20260925 15:24:50.610357 26358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.252:46311 every 8 connection(s)
I20260925 15:24:50.610811 26291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/minicluster-data/master-2/data/info.pb
I20260925 15:24:50.611981 26359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:50.613680 26359 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.211.254" port: 46873 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:24:50.617290 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 26291
I20260925 15:24:50.617394 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790349610595785-3919-0/minicluster-data/master-2/wal/instance
I20260925 15:24:50.617671 3919 external_mini_cluster.cc:1428] Setting key e052259cfa259404f3a92f43d9266111
Sep 25 15:24:50 dist-test-slave-4bk3 krb5kdc[26128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349890, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.252@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
Sep 25 15:24:50 dist-test-slave-4bk3 krb5kdc[26128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.211.252@KRBTEST.COM
I20260925 15:24:50.623540 26212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:50.614501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56752 (local address 127.3.211.254:46873)
0925 15:24:50.614570 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:50.614572 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:50.614794 (+ 222us) server_negotiation.cc:407] Connection header received
0925 15:24:50.615311 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:50.615311 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:50.615323 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:50.615409 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:50.616070 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:50.616647 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:50.618014 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:50.618337 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:50.618433 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:50.620727 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:50.620729 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:50.620729 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:50.620737 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:50.622018 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:50.622452 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:50.622453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:50.622454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:50.622511 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:50.622783 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:50.622783 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:50.622784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:50.622955 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:50.623030 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:50.623342 (+ 312us) server_negotiation.cc:299] Negotiation successful
0925 15:24:50.623432 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 15:24:50.623803 26361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:50.614457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:46873 (local address 127.0.0.1:56752)
0925 15:24:50.614652 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:50.614698 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:50.615198 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:50.615468 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:50.615472 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:50.615731 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:24:50.615960 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:50.615964 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:50.616764 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:50.616765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:50.617923 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:50.617927 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:50.618454 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:50.618455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:50.618613 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:50.619140 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:24:50.619151 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:24:50.620627 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:24:50.622110 (+ 1483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:50.622117 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:50.622120 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:50.622362 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:50.622587 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:50.622588 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:50.622588 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:50.622704 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:50.623054 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:24:50.623059 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:24:50.623186 (+ 127us) client_negotiation.cc:769] Sending connection context
0925 15:24:50.623331 (+ 145us) client_negotiation.cc:240] Negotiation successful
0925 15:24:50.623526 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":118,"threads_started":1}
I20260925 15:24:50.624437 26359 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42085 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:24:50.627138 26363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:50.618149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43058 (local address 127.3.211.252:46311)
0925 15:24:50.618367 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:50.618372 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:50.618400 (+ 28us) server_negotiation.cc:407] Connection header received
0925 15:24:50.618424 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:50.618426 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:50.618448 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:50.618526 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:50.619315 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:50.619976 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:50.620823 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:50.621265 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:50.621359 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:50.624148 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:50.624154 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:50.624155 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:50.624162 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:50.625888 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:50.626282 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:50.626285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:50.626286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:50.626347 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:50.626549 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:50.626550 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:50.626551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:50.626699 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:50.626862 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:50.626945 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 15:24:50.627028 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":131,"threads_started":1}
Sep 25 15:24:50 dist-test-slave-4bk3 krb5kdc[26128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349890, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.252@KRBTEST.COM for kudu/127.3.211.253@KRBTEST.COM
I20260925 15:24:50.632223 26361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:50.624813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.253:42085 (local address 127.0.0.1:41948)
0925 15:24:50.624866 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:50.624876 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:50.624911 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:50.625086 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:50.625087 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:50.625216 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:24:50.625269 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:50.625270 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:50.625944 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:50.625945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:50.626427 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:50.626428 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:50.626835 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:50.626836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:50.626914 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:50.627330 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:24:50.627334 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:24:50.628716 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:24:50.630978 (+ 2262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:50.630979 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:50.630980 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:50.631236 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:50.631496 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:50.631497 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:50.631498 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:50.631572 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:50.631874 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:24:50.631875 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:24:50.631925 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 15:24:50.632001 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 15:24:50.632119 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 15:24:50.632675 26288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:50.624899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41948 (local address 127.3.211.253:42085)
0925 15:24:50.624954 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:50.624956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:50.624964 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:24:50.624988 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:50.624988 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:50.625000 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:50.625083 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:50.625367 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:50.625859 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:50.626512 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:50.626741 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:50.626838 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:50.628811 (+ 1973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:50.628813 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:50.628814 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:50.628821 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:50.630851 (+ 2030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:50.631338 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:50.631339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:50.631340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:50.631401 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:50.631664 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:50.631666 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:50.631666 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:50.631794 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:50.631871 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:50.632140 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 15:24:50.632323 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
Sep 25 15:24:50 dist-test-slave-4bk3 krb5kdc[26128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:24:50.633095 26359 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.211.252" port: 46311 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:24:50.637120 26212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:50.628197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56762 (local address 127.3.211.254:46873)
0925 15:24:50.628246 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:50.628248 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:50.628256 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:24:50.628279 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:50.628280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:50.628289 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:50.628395 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:50.628981 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:50.629562 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:50.630286 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:50.630497 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:50.630572 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:50.633270 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:50.633272 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:50.633272 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:50.633281 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:50.634999 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:50.635436 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:50.635437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:50.635438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:50.635495 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:50.635805 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:50.635810 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:50.635810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:50.636217 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:50.636538 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:50.636889 (+ 351us) server_negotiation.cc:299] Negotiation successful
0925 15:24:50.637001 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Sep 25 15:24:50 dist-test-slave-4bk3 krb5kdc[26128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349890, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.252@KRBTEST.COM for kudu/127.3.211.252@KRBTEST.COM
I20260925 15:24:50.641532 26361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:50.633409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.252:46311 (local address 127.0.0.1:43066)
0925 15:24:50.633464 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:50.633476 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:50.633521 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:50.633650 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:50.633651 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:50.633822 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:24:50.633876 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:50.633878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:50.634813 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:50.634814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:50.635297 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:50.635298 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:50.635853 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:50.635855 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:50.635955 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:50.636725 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:24:50.636730 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:24:50.638587 (+ 1857us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:24:50.640429 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:50.640430 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:50.640431 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:50.640597 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:50.640814 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:50.640815 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:50.640816 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:50.640879 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:50.641137 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:24:50.641138 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:24:50.641175 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 15:24:50.641258 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 15:24:50.641383 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":1}
I20260925 15:24:50.641607 26363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:50.633482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43066 (local address 127.3.211.252:46311)
0925 15:24:50.633523 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:50.633526 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:50.633534 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:24:50.633579 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:50.633580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:50.633594 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:50.633643 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:50.633957 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:50.634714 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:50.635373 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:50.635726 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:50.635789 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:50.638685 (+ 2896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:50.638687 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:50.638688 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:50.638696 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:50.640287 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:50.640689 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:50.640690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:50.640691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:50.640753 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:50.640938 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:50.640938 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:50.640939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:50.641072 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:50.641139 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:50.641289 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 15:24:50.641454 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":86,"server-negotiator.queue_time_us":18}
Sep 25 15:24:50 dist-test-slave-4bk3 krb5kdc[26128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.211.253@KRBTEST.COM
I20260925 15:24:50.643283 26359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9a51cdae15b4a31aeae568604b23f75: Bootstrap starting.
I20260925 15:24:50.644121 26359 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9a51cdae15b4a31aeae568604b23f75: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:50.644513 26359 log.cc:824] T 00000000000000000000000000000000 P a9a51cdae15b4a31aeae568604b23f75: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:50.645287 26359 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9a51cdae15b4a31aeae568604b23f75: No bootstrap required, opened a new log
I20260925 15:24:50.646404 26288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:50.637998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41950 (local address 127.3.211.253:42085)
0925 15:24:50.638049 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:50.638051 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:50.638349 (+ 298us) server_negotiation.cc:407] Connection header received
0925 15:24:50.638431 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:50.638433 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:50.638444 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:50.638501 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:50.639107 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:50.639736 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:50.640663 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:50.640972 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:50.641024 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:50.643608 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:50.643610 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:50.643610 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:50.643618 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:50.645191 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:50.645621 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:50.645622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:50.645623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:50.645678 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:50.645883 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:50.645884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:50.645885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:50.646009 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:50.646186 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:50.646223 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 15:24:50.646296 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 15:24:50.646876 26359 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9a51cdae15b4a31aeae568604b23f75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f3f8b7fe1834d6299ee852a55932f15" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 46873 } } peers { permanent_uuid: "4b752c0b1bf849a29b6bfce9150f1bd3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42085 } } peers { permanent_uuid: "a9a51cdae15b4a31aeae568604b23f75" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 46311 } }
I20260925 15:24:50.647065 26359 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9a51cdae15b4a31aeae568604b23f75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9a51cdae15b4a31aeae568604b23f75, State: Initialized, Role: FOLLOWER
I20260925 15:24:50.647184 26359 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9a51cdae15b4a31aeae568604b23f75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f3f8b7fe1834d6299ee852a55932f15" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 46873 } } peers { permanent_uuid: "4b752c0b1bf849a29b6bfce9150f1bd3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42085 } } peers { permanent_uuid: "a9a51cdae15b4a31aeae568604b23f75" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 46311 } }
I20260925 15:24:50.647547 26359 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9a51cdae15b4a31aeae568604b23f75 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:24:50.647495 26365 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9a51cdae15b4a31aeae568604b23f75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f3f8b7fe1834d6299ee852a55932f15" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 46873 } } peers { permanent_uuid: "4b752c0b1bf849a29b6bfce9150f1bd3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42085 } } peers { permanent_uuid: "a9a51cdae15b4a31aeae568604b23f75" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 46311 } } }
I20260925 15:24:50.647619 26365 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9a51cdae15b4a31aeae568604b23f75 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:24:50.648176 3919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260925 15:24:50.649542 26376 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a9a51cdae15b4a31aeae568604b23f75: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:24:50.649592 26376 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:24:50.649312 26183 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "7f3f8b7fe1834d6299ee852a55932f15"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56762
I20260925 15:24:50.649812 26291 master_runner.cc:186] Error getting master registration for 127.3.211.254:46873: OK, instance_id { permanent_uuid: "7f3f8b7fe1834d6299ee852a55932f15" instance_seqno: 1790349890249040 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 15:24:50.652707 26291 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 15:24:50.661608 26284 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.211.252" port: 46311 } attempt: 3
Sep 25 15:24:50 dist-test-slave-4bk3 krb5kdc[26128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349890, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.253@KRBTEST.COM for kudu/127.3.211.252@KRBTEST.COM
I20260925 15:24:50.670708 26363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:50.662127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43070 (local address 127.3.211.252:46311)
0925 15:24:50.662221 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:50.662223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:50.662231 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:24:50.662258 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:50.662260 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:50.662270 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:50.662323 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:50.662728 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:50.663486 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:50.665094 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:50.665411 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:50.665489 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:50.667775 (+ 2286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:50.667777 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:50.667777 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:50.667786 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:50.669435 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:50.669845 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:50.669846 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:50.669846 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:50.669900 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:50.670098 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:50.670099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:50.670100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:50.670298 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:50.670363 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:50.670527 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 15:24:50.670603 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 15:24:50.670719 26286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:50.662023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.252:46311 (local address 127.0.0.1:43070)
0925 15:24:50.662088 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:50.662100 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:50.662144 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:50.662371 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:50.662372 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:50.662568 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:24:50.662624 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:50.662626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:50.663580 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:50.663581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:50.664995 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:50.664997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:50.665498 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:50.665499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:50.665556 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:50.666084 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:24:50.666090 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:24:50.667671 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:24:50.669526 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:50.669528 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:50.669529 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:50.669763 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:50.669968 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:50.669969 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:50.669970 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:50.670035 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:50.670368 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:24:50.670369 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:24:50.670420 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 15:24:50.670496 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 15:24:50.670613 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 15:24:50.672413 26284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b752c0b1bf849a29b6bfce9150f1bd3: Bootstrap starting.
I20260925 15:24:50.673043 26284 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b752c0b1bf849a29b6bfce9150f1bd3: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:50.673337 26284 log.cc:824] T 00000000000000000000000000000000 P 4b752c0b1bf849a29b6bfce9150f1bd3: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:50.674075 26284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b752c0b1bf849a29b6bfce9150f1bd3: No bootstrap required, opened a new log
I20260925 15:24:50.675519 26284 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b752c0b1bf849a29b6bfce9150f1bd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f3f8b7fe1834d6299ee852a55932f15" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 46873 } } peers { permanent_uuid: "4b752c0b1bf849a29b6bfce9150f1bd3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42085 } } peers { permanent_uuid: "a9a51cdae15b4a31aeae568604b23f75" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 46311 } }
I20260925 15:24:50.675663 26284 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b752c0b1bf849a29b6bfce9150f1bd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b752c0b1bf849a29b6bfce9150f1bd3, State: Initialized, Role: FOLLOWER
I20260925 15:24:50.675757 26284 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b752c0b1bf849a29b6bfce9150f1bd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f3f8b7fe1834d6299ee852a55932f15" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 46873 } } peers { permanent_uuid: "4b752c0b1bf849a29b6bfce9150f1bd3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42085 } } peers { permanent_uuid: "a9a51cdae15b4a31aeae568604b23f75" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 46311 } }
I20260925 15:24:50.676030 26379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b752c0b1bf849a29b6bfce9150f1bd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f3f8b7fe1834d6299ee852a55932f15" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 46873 } } peers { permanent_uuid: "4b752c0b1bf849a29b6bfce9150f1bd3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42085 } } peers { permanent_uuid: "a9a51cdae15b4a31aeae568604b23f75" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 46311 } } }
I20260925 15:24:50.676105 26284 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b752c0b1bf849a29b6bfce9150f1bd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:24:50.676121 26379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b752c0b1bf849a29b6bfce9150f1bd3 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 15:24:50.677887 26390 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4b752c0b1bf849a29b6bfce9150f1bd3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:24:50.677954 26390 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:24:50.678323 26216 master_runner.cc:186] Error getting master registration for 127.3.211.254:46873: OK, instance_id { permanent_uuid: "7f3f8b7fe1834d6299ee852a55932f15" instance_seqno: 1790349890249040 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 15:24:50.681334 26216 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 15:24:50.695647 26209 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.211.252" port: 46311 } attempt: 3
Sep 25 15:24:50 dist-test-slave-4bk3 krb5kdc[26128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349890, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.254@KRBTEST.COM for kudu/127.3.211.252@KRBTEST.COM
I20260925 15:24:50.704656 26363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:50.696250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43078 (local address 127.3.211.252:46311)
0925 15:24:50.696303 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:50.696305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:50.696315 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:24:50.696503 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:50.696504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:50.696517 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:50.696598 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:50.696990 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:50.697737 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:50.698808 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:50.699120 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:50.699211 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:50.701638 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:50.701640 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:50.701640 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:50.701649 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:50.703344 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:50.703794 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:50.703796 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:50.703796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:50.703860 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:50.704069 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:50.704070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:50.704070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:50.704218 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:50.704284 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:50.704467 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 15:24:50.704552 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 15:24:50.704662 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:50.696039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.252:46311 (local address 127.0.0.1:43078)
0925 15:24:50.696104 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:50.696117 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:50.696161 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:50.696621 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:50.696622 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:50.696845 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:24:50.696910 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:50.696913 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:50.697855 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:50.697857 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:50.698702 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:50.698705 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:50.699236 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:50.699238 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:50.699306 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:50.699873 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:24:50.699879 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:24:50.701511 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:24:50.703438 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:50.703440 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:50.703441 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:50.703706 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:50.703940 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:50.703940 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:50.703941 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:50.704004 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:50.704275 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:24:50.704276 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:24:50.704358 (+ 82us) client_negotiation.cc:769] Sending connection context
0925 15:24:50.704440 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 15:24:50.704553 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 15:24:50.706945 26209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f3f8b7fe1834d6299ee852a55932f15: Bootstrap starting.
I20260925 15:24:50.707792 26209 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f3f8b7fe1834d6299ee852a55932f15: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:50.708107 26209 log.cc:824] T 00000000000000000000000000000000 P 7f3f8b7fe1834d6299ee852a55932f15: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:50.709189 26209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f3f8b7fe1834d6299ee852a55932f15: No bootstrap required, opened a new log
I20260925 15:24:50.710639 26209 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f3f8b7fe1834d6299ee852a55932f15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f3f8b7fe1834d6299ee852a55932f15" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 46873 } } peers { permanent_uuid: "4b752c0b1bf849a29b6bfce9150f1bd3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42085 } } peers { permanent_uuid: "a9a51cdae15b4a31aeae568604b23f75" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 46311 } }
I20260925 15:24:50.710807 26209 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f3f8b7fe1834d6299ee852a55932f15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f3f8b7fe1834d6299ee852a55932f15, State: Initialized, Role: FOLLOWER
I20260925 15:24:50.710906 26209 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f3f8b7fe1834d6299ee852a55932f15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f3f8b7fe1834d6299ee852a55932f15" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 46873 } } peers { permanent_uuid: "4b752c0b1bf849a29b6bfce9150f1bd3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42085 } } peers { permanent_uuid: "a9a51cdae15b4a31aeae568604b23f75" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 46311 } }
I20260925 15:24:50.711300 26183 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f3f8b7fe1834d6299ee852a55932f15 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 15:24:50.711418 26183 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f3f8b7fe1834d6299ee852a55932f15 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:50.711308 26393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f3f8b7fe1834d6299ee852a55932f15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f3f8b7fe1834d6299ee852a55932f15" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 46873 } } peers { permanent_uuid: "4b752c0b1bf849a29b6bfce9150f1bd3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42085 } } peers { permanent_uuid: "a9a51cdae15b4a31aeae568604b23f75" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 46311 } } }
I20260925 15:24:50.711601 26393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f3f8b7fe1834d6299ee852a55932f15 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:24:50.711285 26209 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f3f8b7fe1834d6299ee852a55932f15 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:24:50.712384 26183 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f3f8b7fe1834d6299ee852a55932f15 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f3f8b7fe1834d6299ee852a55932f15" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 46873 } } peers { permanent_uuid: "4b752c0b1bf849a29b6bfce9150f1bd3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42085 } } peers { permanent_uuid: "a9a51cdae15b4a31aeae568604b23f75" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 46311 } }
I20260925 15:24:50.712729 26183 leader_election.cc:288] T 00000000000000000000000000000000 P 7f3f8b7fe1834d6299ee852a55932f15 [CANDIDATE]: Term 1 election: Requested vote from peers 4b752c0b1bf849a29b6bfce9150f1bd3 (127.3.211.253:42085), a9a51cdae15b4a31aeae568604b23f75 (127.3.211.252:46311)
I20260925 15:24:50.713165 26258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7f3f8b7fe1834d6299ee852a55932f15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4b752c0b1bf849a29b6bfce9150f1bd3"
I20260925 15:24:50.713323 26258 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b752c0b1bf849a29b6bfce9150f1bd3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:50.714286 26258 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4b752c0b1bf849a29b6bfce9150f1bd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f3f8b7fe1834d6299ee852a55932f15 in term 1.
I20260925 15:24:50.714401 26333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7f3f8b7fe1834d6299ee852a55932f15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a9a51cdae15b4a31aeae568604b23f75"
I20260925 15:24:50.714543 26333 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9a51cdae15b4a31aeae568604b23f75 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:50.714745 26159 leader_election.cc:302] T 00000000000000000000000000000000 P 7f3f8b7fe1834d6299ee852a55932f15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b752c0b1bf849a29b6bfce9150f1bd3, 7f3f8b7fe1834d6299ee852a55932f15; no voters:
I20260925 15:24:50.715554 26333 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a9a51cdae15b4a31aeae568604b23f75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f3f8b7fe1834d6299ee852a55932f15 in term 1.
I20260925 15:24:50.716532 26393 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f3f8b7fe1834d6299ee852a55932f15 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:24:50.716756 26393 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f3f8b7fe1834d6299ee852a55932f15 [term 1 LEADER]: Becoming Leader. State: Replica: 7f3f8b7fe1834d6299ee852a55932f15, State: Running, Role: LEADER
I20260925 15:24:50.716920 26393 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f3f8b7fe1834d6299ee852a55932f15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f3f8b7fe1834d6299ee852a55932f15" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 46873 } } peers { permanent_uuid: "4b752c0b1bf849a29b6bfce9150f1bd3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42085 } } peers { permanent_uuid: "a9a51cdae15b4a31aeae568604b23f75" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 46311 } }
W20260925 15:24:50.717247 26408 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7f3f8b7fe1834d6299ee852a55932f15: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:24:50.717350 26408 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:24:50.717593 26393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f3f8b7fe1834d6299ee852a55932f15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f3f8b7fe1834d6299ee852a55932f15. Latest consensus state: current_term: 1 leader_uuid: "7f3f8b7fe1834d6299ee852a55932f15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f3f8b7fe1834d6299ee852a55932f15" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 46873 } } peers { permanent_uuid: "4b752c0b1bf849a29b6bfce9150f1bd3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42085 } } peers { permanent_uuid: "a9a51cdae15b4a31aeae568604b23f75" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 46311 } } }
I20260925 15:24:50.717732 26393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f3f8b7fe1834d6299ee852a55932f15 [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:50.728499 26412 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:24:50.728752 26412 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:24:50.729204 26212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:50.720637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56778 (local address 127.3.211.254:46873)
0925 15:24:50.720727 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:50.720730 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:50.720974 (+ 244us) server_negotiation.cc:407] Connection header received
0925 15:24:50.721078 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:50.721079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:50.721091 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:50.721121 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:50.721910 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:50.722644 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:50.723691 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:50.724002 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:50.724041 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:50.725871 (+ 1830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:50.725873 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:50.725873 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:50.725880 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:50.727693 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:50.728238 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:50.728239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:50.728240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:50.728296 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:50.728545 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:50.728546 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:50.728546 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:50.728673 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:50.728744 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:50.729037 (+ 293us) server_negotiation.cc:299] Negotiation successful
0925 15:24:50.729123 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 15:24:50.729249 26363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:50.720631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43090 (local address 127.3.211.252:46311)
0925 15:24:50.720690 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:50.720693 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:50.720864 (+ 171us) server_negotiation.cc:407] Connection header received
0925 15:24:50.720965 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:50.720966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:50.720976 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:50.721007 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:50.721818 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:50.722568 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:50.723666 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:50.723971 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:50.724012 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:50.725815 (+ 1803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:50.725816 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:50.725817 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:50.725824 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:50.727698 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:50.728218 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:50.728219 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:50.728219 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:50.728274 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:50.728504 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:50.728505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:50.728505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:50.728641 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:50.728713 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:50.729079 (+ 366us) server_negotiation.cc:299] Negotiation successful
0925 15:24:50.729158 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 15:24:50.730911 26333 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a9a51cdae15b4a31aeae568604b23f75 [term 1 FOLLOWER]: Refusing update from remote peer 7f3f8b7fe1834d6299ee852a55932f15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:24:50.730911 26258 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4b752c0b1bf849a29b6bfce9150f1bd3 [term 1 FOLLOWER]: Refusing update from remote peer 7f3f8b7fe1834d6299ee852a55932f15: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:24:50.731237 26393 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7f3f8b7fe1834d6299ee852a55932f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b752c0b1bf849a29b6bfce9150f1bd3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:24:50.731344 26410 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7f3f8b7fe1834d6299ee852a55932f15 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9a51cdae15b4a31aeae568604b23f75" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 46311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:24:50.732575 26379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b752c0b1bf849a29b6bfce9150f1bd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f3f8b7fe1834d6299ee852a55932f15. Latest consensus state: current_term: 1 leader_uuid: "7f3f8b7fe1834d6299ee852a55932f15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f3f8b7fe1834d6299ee852a55932f15" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 46873 } } peers { permanent_uuid: "4b752c0b1bf849a29b6bfce9150f1bd3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42085 } } peers { permanent_uuid: "a9a51cdae15b4a31aeae568604b23f75" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 46311 } } }
I20260925 15:24:50.732674 26379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b752c0b1bf849a29b6bfce9150f1bd3 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:24:50.732738 26365 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9a51cdae15b4a31aeae568604b23f75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f3f8b7fe1834d6299ee852a55932f15. Latest consensus state: current_term: 1 leader_uuid: "7f3f8b7fe1834d6299ee852a55932f15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f3f8b7fe1834d6299ee852a55932f15" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 46873 } } peers { permanent_uuid: "4b752c0b1bf849a29b6bfce9150f1bd3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42085 } } peers { permanent_uuid: "a9a51cdae15b4a31aeae568604b23f75" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 46311 } } }
I20260925 15:24:50.732807 26365 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9a51cdae15b4a31aeae568604b23f75 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:24:50.733735 26393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f3f8b7fe1834d6299ee852a55932f15 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7f3f8b7fe1834d6299ee852a55932f15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f3f8b7fe1834d6299ee852a55932f15" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 46873 } } peers { permanent_uuid: "4b752c0b1bf849a29b6bfce9150f1bd3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42085 } } peers { permanent_uuid: "a9a51cdae15b4a31aeae568604b23f75" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 46311 } } }
I20260925 15:24:50.733832 26393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f3f8b7fe1834d6299ee852a55932f15 [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:50.734215 26365 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9a51cdae15b4a31aeae568604b23f75 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7f3f8b7fe1834d6299ee852a55932f15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f3f8b7fe1834d6299ee852a55932f15" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 46873 } } peers { permanent_uuid: "4b752c0b1bf849a29b6bfce9150f1bd3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42085 } } peers { permanent_uuid: "a9a51cdae15b4a31aeae568604b23f75" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 46311 } } }
I20260925 15:24:50.734304 26365 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9a51cdae15b4a31aeae568604b23f75 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:24:50.734498 26416 mvcc.cc:205] Tried to move back new op lower bound from 7333273152430911488 to 7333273152377716736. Current Snapshot: MvccSnapshot[applied={T|T < 7333273152430911488}]
I20260925 15:24:50.735019 26417 mvcc.cc:205] Tried to move back new op lower bound from 7333273152430911488 to 7333273152377716736. Current Snapshot: MvccSnapshot[applied={T|T < 7333273152430911488}]
I20260925 15:24:50.736178 26410 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f3f8b7fe1834d6299ee852a55932f15 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7f3f8b7fe1834d6299ee852a55932f15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f3f8b7fe1834d6299ee852a55932f15" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 46873 } } peers { permanent_uuid: "4b752c0b1bf849a29b6bfce9150f1bd3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42085 } } peers { permanent_uuid: "a9a51cdae15b4a31aeae568604b23f75" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 46311 } } }
I20260925 15:24:50.736275 26410 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f3f8b7fe1834d6299ee852a55932f15 [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:50.736725 26379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b752c0b1bf849a29b6bfce9150f1bd3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7f3f8b7fe1834d6299ee852a55932f15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f3f8b7fe1834d6299ee852a55932f15" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 46873 } } peers { permanent_uuid: "4b752c0b1bf849a29b6bfce9150f1bd3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 42085 } } peers { permanent_uuid: "a9a51cdae15b4a31aeae568604b23f75" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 46311 } } }
I20260925 15:24:50.736810 26379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b752c0b1bf849a29b6bfce9150f1bd3 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:24:50.736979 26412 catalog_manager.cc:1347] Generated new cluster ID: fd2467626d614bf09cdf650e8f65c056
I20260925 15:24:50.737032 26412 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:24:50.738705 26288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:50.720924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41964 (local address 127.3.211.253:42085)
0925 15:24:50.720964 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:50.720966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:50.729517 (+ 8551us) server_negotiation.cc:407] Connection header received
0925 15:24:50.730003 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:50.730005 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:50.730020 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:50.730074 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:50.730739 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:50.731509 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:50.732655 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:50.732954 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:50.732999 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:50.734863 (+ 1864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:50.734865 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:50.734865 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:50.734873 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:50.736777 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:50.737372 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:50.737373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:50.737374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:50.737434 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:50.737648 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:50.737649 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:50.737649 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:50.737994 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:50.738227 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:50.738510 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 15:24:50.738604 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 15:24:50.740427 26418 mvcc.cc:205] Tried to move back new op lower bound from 7333273152430911488 to 7333273152377716736. Current Snapshot: MvccSnapshot[applied={T|T < 7333273152430911488}]
I20260925 15:24:50.744240 26412 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:24:50.745012 26412 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:24:50.759220 26412 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f3f8b7fe1834d6299ee852a55932f15: Generated new TSK 0
I20260925 15:24:50.759575 26412 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 15:24:50.788964 26288 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:24:50.787270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41972 (local address 127.3.211.253:42085)
0925 15:24:50.787318 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:50.787320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:50.788232 (+ 912us) server_negotiation.cc:407] Connection header received
0925 15:24:50.788358 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:50.788360 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:50.788371 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:50.788420 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:50.788884 (+ 464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41972: BlockingRecv error: recv got EOF from 127.0.0.1:41972 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260925 15:24:50.789717 26212 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:24:50.787654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56792 (local address 127.3.211.254:46873)
0925 15:24:50.787694 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:50.787696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:50.789204 (+ 1508us) server_negotiation.cc:407] Connection header received
0925 15:24:50.789301 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:50.789302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:50.789313 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:50.789343 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:50.789656 (+ 313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56792: BlockingRecv error: recv got EOF from 127.0.0.1:56792 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 15:24:50.790421 26363 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:24:50.787976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43098 (local address 127.3.211.252:46311)
0925 15:24:50.788016 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:50.788018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:50.789926 (+ 1908us) server_negotiation.cc:407] Connection header received
0925 15:24:50.790021 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:50.790022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:50.790033 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:50.790062 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:50.790347 (+ 285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43098: BlockingRecv error: recv got EOF from 127.0.0.1:43098 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 15:24:50.792984 26288 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:24:50.791739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41986 (local address 127.3.211.253:42085)
0925 15:24:50.791776 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:50.791777 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:50.792418 (+ 641us) server_negotiation.cc:407] Connection header received
0925 15:24:50.792532 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:50.792533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:50.792553 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:50.792582 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:50.792929 (+ 347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41986: BlockingRecv error: recv got EOF from 127.0.0.1:41986 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 15:24:50.796168 26288 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:24:50.795271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42002 (local address 127.3.211.253:42085)
0925 15:24:50.795310 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:50.795312 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:50.795589 (+ 277us) server_negotiation.cc:407] Connection header received
0925 15:24:50.795691 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:50.795692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:50.795702 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:50.795735 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:50.796109 (+ 374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42002: BlockingRecv error: recv got EOF from 127.0.0.1:42002 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 15:24:50.800455 26288 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:24:50.799509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42010 (local address 127.3.211.253:42085)
0925 15:24:50.799546 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:50.799548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:50.799806 (+ 258us) server_negotiation.cc:407] Connection header received
0925 15:24:50.799911 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:50.799912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:50.799922 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:50.799952 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:50.800308 (+ 356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42010: BlockingRecv error: recv got EOF from 127.0.0.1:42010 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260925 15:24:50.806643 26288 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:24:50.805735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42022 (local address 127.3.211.253:42085)
0925 15:24:50.805780 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:50.805782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:50.806053 (+ 271us) server_negotiation.cc:407] Connection header received
0925 15:24:50.806156 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:50.806157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:50.806168 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:50.806198 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:50.806579 (+ 381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42022: BlockingRecv error: recv got EOF from 127.0.0.1:42022 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 15:24:50.817221 26288 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:24:50.816137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42028 (local address 127.3.211.253:42085)
0925 15:24:50.816185 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:50.816198 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:50.816537 (+ 339us) server_negotiation.cc:407] Connection header received
0925 15:24:50.816658 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:50.816659 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:50.816672 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:50.816708 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:50.817139 (+ 431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42028: BlockingRecv error: recv got EOF from 127.0.0.1:42028 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260925 15:24:50.836661 26288 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:24:50.835640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42038 (local address 127.3.211.253:42085)
0925 15:24:50.835695 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:50.835698 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:50.836003 (+ 305us) server_negotiation.cc:407] Connection header received
0925 15:24:50.836114 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:50.836115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:50.836128 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:50.836159 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:50.836603 (+ 444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42038: BlockingRecv error: recv got EOF from 127.0.0.1:42038 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260925 15:24:50.870824 26288 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:24:50.870043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42048 (local address 127.3.211.253:42085)
0925 15:24:50.870092 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:50.870095 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:50.870213 (+ 118us) server_negotiation.cc:407] Connection header received
0925 15:24:50.870267 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:50.870268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:50.870280 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:50.870362 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:50.870765 (+ 403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42048: BlockingRecv error: recv got EOF from 127.0.0.1:42048 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260925 15:24:50.937143 26288 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:24:50.936251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42064 (local address 127.3.211.253:42085)
0925 15:24:50.936321 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:50.936323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:50.936512 (+ 189us) server_negotiation.cc:407] Connection header received
0925 15:24:50.936570 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:50.936571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:50.936581 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:50.936624 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:50.937065 (+ 441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42064: BlockingRecv error: recv got EOF from 127.0.0.1:42064 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260925 15:24:51.067279 26288 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:24:51.066583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42068 (local address 127.3.211.253:42085)
0925 15:24:51.066636 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:51.066638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:51.066705 (+ 67us) server_negotiation.cc:407] Connection header received
0925 15:24:51.066747 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:51.066747 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:51.066758 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:51.066811 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:51.067192 (+ 381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42068: BlockingRecv error: recv got EOF from 127.0.0.1:42068 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 15:24:51.325197 26288 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:24:51.324658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42076 (local address 127.3.211.253:42085)
0925 15:24:51.324718 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:51.324719 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:51.324730 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:24:51.324775 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:51.324776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:51.324786 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:51.324834 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:51.325148 (+ 314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42076: BlockingRecv error: recv got EOF from 127.0.0.1:42076 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 15:24:51.650084 26376 catalog_manager.cc:1259] Loaded cluster ID: fd2467626d614bf09cdf650e8f65c056
I20260925 15:24:51.650202 26376 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a9a51cdae15b4a31aeae568604b23f75: loading cluster ID for follower catalog manager: success
I20260925 15:24:51.651033 26376 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a9a51cdae15b4a31aeae568604b23f75: acquiring CA information for follower catalog manager: success
I20260925 15:24:51.651367 26376 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a9a51cdae15b4a31aeae568604b23f75: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 15:24:51.678458 26390 catalog_manager.cc:1259] Loaded cluster ID: fd2467626d614bf09cdf650e8f65c056
I20260925 15:24:51.678580 26390 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4b752c0b1bf849a29b6bfce9150f1bd3: loading cluster ID for follower catalog manager: success
I20260925 15:24:51.679401 26390 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4b752c0b1bf849a29b6bfce9150f1bd3: acquiring CA information for follower catalog manager: success
I20260925 15:24:51.679864 26390 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4b752c0b1bf849a29b6bfce9150f1bd3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 15:24:51.841156 26489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:51.838466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42078 (local address 127.3.211.253:42085)
0925 15:24:51.838624 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:51.838625 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:51.838641 (+ 16us) server_negotiation.cc:407] Connection header received
0925 15:24:51.838672 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:51.838672 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:51.838731 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:51.838810 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:24:51.838998 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:51.839513 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:51.840358 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:51.840574 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:51.840638 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:51.840851 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:24:51.840969 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:24:51.841029 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:51.841070 (+ 41us) server_negotiation.cc:299] Negotiation successful
0925 15:24:51.841089 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":62,"threads_started":1}
I20260925 15:24:51.845731 26495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:51.843098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43114 (local address 127.3.211.252:46311)
0925 15:24:51.843222 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:51.843224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:51.843236 (+ 12us) server_negotiation.cc:407] Connection header received
0925 15:24:51.843351 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:51.843352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:51.843399 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:51.843499 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:24:51.843628 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:51.844109 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:51.844930 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:51.845140 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:51.845216 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:51.845384 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:24:51.845501 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:24:51.845592 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:51.845615 (+ 23us) server_negotiation.cc:299] Negotiation successful
0925 15:24:51.845637 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":52,"threads_started":1}
I20260925 15:24:51.850534 26495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:51.847620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43120 (local address 127.3.211.252:46311)
0925 15:24:51.847681 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:51.847683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:51.848061 (+ 378us) server_negotiation.cc:407] Connection header received
0925 15:24:51.848217 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:51.848218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:51.848230 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:51.848259 (+ 29us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:24:51.848455 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:51.849013 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:51.849867 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:51.850072 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:51.850151 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:51.850300 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:24:51.850336 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:24:51.850375 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:51.850456 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 15:24:51.850474 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 15:24:51.851385 26489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:51.847628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42084 (local address 127.3.211.253:42085)
0925 15:24:51.847795 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:51.847797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:51.847908 (+ 111us) server_negotiation.cc:407] Connection header received
0925 15:24:51.847980 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:51.847982 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:51.847996 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:51.848049 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:24:51.848247 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:51.849019 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:51.850227 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:51.850548 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:51.850621 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:51.851077 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:24:51.851112 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:24:51.851187 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:51.851268 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 15:24:51.851298 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143}
I20260925 15:24:51.851799 26503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:51.847687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56808 (local address 127.3.211.254:46873)
0925 15:24:51.847815 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:51.847817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:51.847824 (+ 7us) server_negotiation.cc:407] Connection header received
0925 15:24:51.847859 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:51.847860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:51.847902 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:51.848011 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:24:51.848153 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:51.849038 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:51.850669 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:51.850992 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:51.851062 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:51.851399 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:24:51.851525 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:24:51.851621 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:51.851683 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 15:24:51.851709 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":59,"threads_started":1}
I20260925 15:24:51.853502 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 26141
I20260925 15:24:51.859127 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 26216
I20260925 15:24:51.864084 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 26291
2026-09-25T15:24:51Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2948 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:24:51 dist-test-slave-4bk3 krb5kdc[26506](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:24:51 dist-test-slave-4bk3 krb5kdc[26506](info): set up 2 sockets
Sep 25 15:24:51 dist-test-slave-4bk3 krb5kdc[26506](info): commencing operation
krb5kdc: starting...
W20260925 15:24:52.908818 3919 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/krb5kdc/test-user.keytab.
Sep 25 15:24:52 dist-test-slave-4bk3 krb5kdc[26506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:24:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:24:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.211.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
Entry for principal kudu/127.3.211.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab.
I20260925 15:24:53.032487 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:37851
--webserver_interface=127.3.211.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:38969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.211.254:37851,127.3.211.253:34867,127.3.211.252:44569
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:24:53.117743 26519 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 15:24:53.117978 26519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:53.117996 26519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:53.118019 26519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:53.119668 26519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:24:53.119758 26519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:53.119788 26519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:53.119833 26519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:24:53.119868 26519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:24:53.121953 26519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:38969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.211.254:37851,127.3.211.253:34867,127.3.211.252:44569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.254.keytab
--principal=kudu/127.3.211.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.254:37851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.211.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:24:53.122507 26519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:53.122814 26519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:53.128062 26525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:53.128065 26527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:53.128077 26524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:53.128293 26519 server_base.cc:1034] running on GCE node
I20260925 15:24:53.128571 26519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:53.128926 26519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:53.130108 26519 hybrid_clock.cc:648] HybridClock initialized: now 1790349893130098 us; error 30 us; skew 500 ppm
Sep 25 15:24:53 dist-test-slave-4bk3 krb5kdc[26506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349893, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:24:53.144941 26519 init.cc:377] Logged in from keytab as kudu/127.3.211.254@KRBTEST.COM (short username kudu)
I20260925 15:24:53.145506 26519 webserver.cc:460] Webserver started at http://127.3.211.254:35151/ using document root <none> and password file <none>
I20260925 15:24:53.145745 26519 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:53.145814 26519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:53.145973 26519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:53.147101 26519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/minicluster-data/master-0/data/instance:
uuid: "aa0a608c86e445c2bf2f0874c8159805"
format_stamp: "Formatted at 2026-09-25 15:24:53 on dist-test-slave-4bk3"
server_key: "664152d973f51c66ad9bde8a69b1e576"
server_key_iv: "f503e896cc8943e0fff27fb7224d1f11"
server_key_version: "encryptionkey@0"
I20260925 15:24:53.147490 26519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/minicluster-data/master-0/wal/instance:
uuid: "aa0a608c86e445c2bf2f0874c8159805"
format_stamp: "Formatted at 2026-09-25 15:24:53 on dist-test-slave-4bk3"
server_key: "664152d973f51c66ad9bde8a69b1e576"
server_key_iv: "f503e896cc8943e0fff27fb7224d1f11"
server_key_version: "encryptionkey@0"
I20260925 15:24:53.149093 26519 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:24:53.150027 26534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:53.150321 26519 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:24:53.150461 26519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/minicluster-data/master-0/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/minicluster-data/master-0/wal
uuid: "aa0a608c86e445c2bf2f0874c8159805"
format_stamp: "Formatted at 2026-09-25 15:24:53 on dist-test-slave-4bk3"
server_key: "664152d973f51c66ad9bde8a69b1e576"
server_key_iv: "f503e896cc8943e0fff27fb7224d1f11"
server_key_version: "encryptionkey@0"
I20260925 15:24:53.150550 26519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:53.170605 26519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:53.173748 26519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:53.174156 26519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:53.180085 26519 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.254:37851
I20260925 15:24:53.180308 26586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.254:37851 every 8 connection(s)
I20260925 15:24:53.180601 26519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/minicluster-data/master-0/data/info.pb
I20260925 15:24:53.181632 26587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:53.183225 26587 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.211.254" port: 37851 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:24:53.188328 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 26519
I20260925 15:24:53.188548 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/minicluster-data/master-0/wal/instance
I20260925 15:24:53.188778 3919 external_mini_cluster.cc:1428] Setting key 4c6b78f359df364c87b1f4a0439bcf5c
Sep 25 15:24:53 dist-test-slave-4bk3 krb5kdc[26506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349893, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.254@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:24:53.195556 26589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:53.183966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:37851 (local address 127.0.0.1:55528)
0925 15:24:53.184133 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:53.184174 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:53.184711 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:53.185067 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:53.185070 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:53.185348 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:24:53.185516 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:53.185519 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:53.186760 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:53.186763 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:53.187801 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:53.187806 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:53.188579 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:53.188600 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:53.188697 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:53.189275 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:24:53.189286 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:24:53.191362 (+ 2076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:24:53.193751 (+ 2389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:53.193759 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:53.193761 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:53.194002 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:53.194347 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:53.194348 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:53.194349 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:53.194459 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:53.195024 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:24:53.195031 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:24:53.195097 (+ 66us) client_negotiation.cc:769] Sending connection context
0925 15:24:53.195189 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 15:24:53.195365 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":74,"threads_started":1}
I20260925 15:24:53.195617 26590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:53.184129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55528 (local address 127.3.211.254:37851)
0925 15:24:53.184264 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:53.184275 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:53.184304 (+ 29us) server_negotiation.cc:407] Connection header received
0925 15:24:53.184882 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:53.184884 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:53.184903 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:53.184998 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:53.185715 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:53.186577 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:53.187926 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:53.188300 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:53.188485 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:53.191468 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:53.191477 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:53.191477 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:53.191486 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:53.193624 (+ 2138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:53.194102 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:53.194107 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:53.194107 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:53.194232 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:53.194538 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:53.194539 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:53.194539 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:53.194906 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:53.195008 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:53.195257 (+ 249us) server_negotiation.cc:299] Negotiation successful
0925 15:24:53.195475 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":97,"server-negotiator.queue_time_us":101,"thread_start_us":83,"threads_started":1}
I20260925 15:24:53.196573 26587 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.211.253" port: 34867 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:24:53.197009 26589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:53.196854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.253:34867 (local address 127.0.0.1:55468)
0925 15:24:53.196933 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:53.196983 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.253:34867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260925 15:24:53.197213 26535 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.211.253:34867: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.3.211.253@KRBTEST.COM; defaulting to no policy
I20260925 15:24:53.197775 26589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:53.197644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.253:34867 (local address 127.0.0.1:55476)
0925 15:24:53.197714 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:53.197743 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.253:34867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 15:24:53.197894 26587 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.211.253:34867: Network error: Client connection negotiation failed: client connection to 127.3.211.253:34867: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.253.keytab.
Entry for principal kudu/127.3.211.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.253.keytab.
I20260925 15:24:53.216810 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.253:34867
--webserver_interface=127.3.211.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.253.keytab
--principal=kudu/127.3.211.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:38969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.211.254:37851,127.3.211.253:34867,127.3.211.252:44569
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:24:53.247170 26587 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.211.253" port: 34867 } attempt: 1
I20260925 15:24:53.247713 26589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:53.247567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.253:34867 (local address 127.0.0.1:55482)
0925 15:24:53.247640 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:53.247684 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.253:34867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 15:24:53.248090 26589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:53.247992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.253:34867 (local address 127.0.0.1:55496)
0925 15:24:53.248039 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:53.248067 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.253:34867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 15:24:53.248179 26587 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.211.253:34867: Network error: Client connection negotiation failed: client connection to 127.3.211.253:34867: connect: Connection refused (error 111)
W20260925 15:24:53.304384 26594 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 15:24:53.304689 26594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:53.304734 26594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:53.304780 26594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:53.306705 26594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:24:53.306874 26594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:53.306905 26594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:53.306929 26594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:24:53.306952 26594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:24:53.309840 26594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:38969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.211.254:37851,127.3.211.253:34867,127.3.211.252:44569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.253.keytab
--principal=kudu/127.3.211.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.253:34867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.211.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:24:53.310214 26594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:53.310519 26594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:53.315639 26602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:53.315644 26599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:53.315639 26600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:53.315946 26594 server_base.cc:1034] running on GCE node
I20260925 15:24:53.316174 26594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:53.316301 26587 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.211.253" port: 34867 } attempt: 2
I20260925 15:24:53.316556 26594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:53.316737 26589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:53.316615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.253:34867 (local address 127.0.0.1:55508)
0925 15:24:53.316675 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:53.316703 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.253:34867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 15:24:53.317080 26589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:53.316992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.253:34867 (local address 127.0.0.1:55524)
0925 15:24:53.317041 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:53.317064 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.253:34867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 15:24:53.317189 26587 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.211.253:34867: Network error: Client connection negotiation failed: client connection to 127.3.211.253:34867: connect: Connection refused (error 111)
I20260925 15:24:53.317739 26594 hybrid_clock.cc:648] HybridClock initialized: now 1790349893317715 us; error 32 us; skew 500 ppm
Sep 25 15:24:53 dist-test-slave-4bk3 krb5kdc[26506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349893, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:24:53.320753 26594 init.cc:377] Logged in from keytab as kudu/127.3.211.253@KRBTEST.COM (short username kudu)
I20260925 15:24:53.321355 26594 webserver.cc:460] Webserver started at http://127.3.211.253:44689/ using document root <none> and password file <none>
I20260925 15:24:53.321666 26594 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:53.321736 26594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:53.321868 26594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:53.323208 26594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/minicluster-data/master-1/data/instance:
uuid: "d77bf57eb9f74582b5e760b46babade3"
format_stamp: "Formatted at 2026-09-25 15:24:53 on dist-test-slave-4bk3"
server_key: "18a85aae020d9e455382663e2cae1312"
server_key_iv: "922edeb80b8f7e2b275e4fef9d11a6b2"
server_key_version: "encryptionkey@0"
I20260925 15:24:53.323539 26594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/minicluster-data/master-1/wal/instance:
uuid: "d77bf57eb9f74582b5e760b46babade3"
format_stamp: "Formatted at 2026-09-25 15:24:53 on dist-test-slave-4bk3"
server_key: "18a85aae020d9e455382663e2cae1312"
server_key_iv: "922edeb80b8f7e2b275e4fef9d11a6b2"
server_key_version: "encryptionkey@0"
I20260925 15:24:53.325089 26594 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:24:53.326226 26609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:53.326515 26594 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:24:53.326620 26594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/minicluster-data/master-1/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/minicluster-data/master-1/wal
uuid: "d77bf57eb9f74582b5e760b46babade3"
format_stamp: "Formatted at 2026-09-25 15:24:53 on dist-test-slave-4bk3"
server_key: "18a85aae020d9e455382663e2cae1312"
server_key_iv: "922edeb80b8f7e2b275e4fef9d11a6b2"
server_key_version: "encryptionkey@0"
I20260925 15:24:53.326713 26594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:53.343214 26594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:53.345438 26594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:53.345559 26594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:53.349825 26594 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.253:34867
I20260925 15:24:53.349864 26661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.253:34867 every 8 connection(s)
I20260925 15:24:53.350229 26594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/minicluster-data/master-1/data/info.pb
I20260925 15:24:53.351274 26662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:53.352919 26662 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.211.254" port: 37851 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:24:53.356853 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 26594
I20260925 15:24:53.356968 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/minicluster-data/master-1/wal/instance
I20260925 15:24:53.357167 3919 external_mini_cluster.cc:1428] Setting key 328270842827b46f79a84c1406843938
Sep 25 15:24:53 dist-test-slave-4bk3 krb5kdc[26506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349893, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.253@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:24:53.363806 26590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:53.353692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55538 (local address 127.3.211.254:37851)
0925 15:24:53.353739 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:53.353741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:53.353973 (+ 232us) server_negotiation.cc:407] Connection header received
0925 15:24:53.354555 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:53.354556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:53.354572 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:53.354620 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:53.355271 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:53.355768 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:53.357001 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:53.357281 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:53.357351 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:53.360170 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:53.360171 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:53.360172 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:53.360185 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:53.362087 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:53.362591 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:53.362592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:53.362593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:53.362661 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:53.362925 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:53.362926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:53.362926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:53.363164 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:53.363299 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:53.363605 (+ 306us) server_negotiation.cc:299] Negotiation successful
0925 15:24:53.363704 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 15:24:53.364022 26664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:53.353671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:37851 (local address 127.0.0.1:55538)
0925 15:24:53.353849 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:53.353885 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:53.354440 (+ 555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:53.354694 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:53.354698 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:53.354957 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:24:53.355154 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:53.355158 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:53.355875 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:53.355876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:53.356871 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:53.356874 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:53.357379 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:53.357380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:53.357543 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:53.358158 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:24:53.358169 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:24:53.360049 (+ 1880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:24:53.362218 (+ 2169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:53.362229 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:53.362232 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:53.362488 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:53.362734 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:53.362735 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:53.362735 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:53.362849 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:53.363291 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:24:53.363299 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:24:53.363458 (+ 159us) client_negotiation.cc:769] Sending connection context
0925 15:24:53.363584 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 15:24:53.363769 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":91,"threads_started":1}
I20260925 15:24:53.364791 26662 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.211.253" port: 34867 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.211.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.211.252@KRBTEST.COM" created.
Sep 25 15:24:53 dist-test-slave-4bk3 krb5kdc[26506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349893, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.253@KRBTEST.COM for kudu/127.3.211.253@KRBTEST.COM
I20260925 15:24:53.374662 26664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:53.365119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.253:34867 (local address 127.0.0.1:55536)
0925 15:24:53.365175 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:53.365186 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:53.365234 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:53.365735 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:53.365736 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:53.365867 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:24:53.365941 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:53.365942 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:53.367344 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:53.367345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:53.367977 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:53.367980 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:53.368629 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:53.368631 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:53.368702 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:53.369407 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:24:53.369412 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:24:53.370686 (+ 1274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:24:53.373101 (+ 2415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:53.373114 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:53.373115 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:53.373304 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:53.373630 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:53.373631 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:53.373631 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:53.373723 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:53.374191 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:24:53.374193 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:24:53.374272 (+ 79us) client_negotiation.cc:769] Sending connection context
0925 15:24:53.374388 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 15:24:53.374556 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 15:24:53.374989 26666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:53.365180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55536 (local address 127.3.211.253:34867)
0925 15:24:53.365395 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:53.365400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:53.365429 (+ 29us) server_negotiation.cc:407] Connection header received
0925 15:24:53.365456 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:53.365459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:53.365474 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:53.365754 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:53.366053 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:53.367245 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:53.368068 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:53.368518 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:53.368583 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:53.370819 (+ 2236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:53.370831 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:53.370831 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:53.370839 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:53.372972 (+ 2133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:53.373420 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:53.373424 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:53.373425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:53.373494 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:53.373801 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:53.373803 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:53.373803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:53.374031 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:53.374131 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:53.374458 (+ 327us) server_negotiation.cc:299] Negotiation successful
0925 15:24:53.374697 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":80,"server-negotiator.queue_time_us":176,"thread_start_us":110,"threads_started":1}
I20260925 15:24:53.375901 26662 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.211.252" port: 44569 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:24:53.376487 26664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:53.376319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.252:44569 (local address 127.0.0.1:48678)
0925 15:24:53.376404 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:53.376462 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.252:44569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260925 15:24:53.376670 26613 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.211.252:44569: connect: Connection refused (error 111)
I20260925 15:24:53.377183 26664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:53.377019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.252:44569 (local address 127.0.0.1:48682)
0925 15:24:53.377130 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:53.377162 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.252:44569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260925 15:24:53.377358 26662 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.211.252:44569: Network error: Client connection negotiation failed: client connection to 127.3.211.252:44569: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.211.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.252.keytab.
Entry for principal kudu/127.3.211.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.252.keytab.
I20260925 15:24:53.389412 3919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskAzNr2f/build/release/bin/kudu
/tmp/dist-test-taskAzNr2f/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.252:44569
--webserver_interface=127.3.211.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.252.keytab
--principal=kudu/127.3.211.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.211.212:38969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.211.254:37851,127.3.211.253:34867,127.3.211.252:44569
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:24:53.408386 26587 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.211.253" port: 34867 } attempt: 3
Sep 25 15:24:53 dist-test-slave-4bk3 krb5kdc[26506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349893, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.254@KRBTEST.COM for kudu/127.3.211.253@KRBTEST.COM
I20260925 15:24:53.415968 26589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:53.408830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.253:34867 (local address 127.0.0.1:55548)
0925 15:24:53.408904 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:53.408914 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:53.408978 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:53.409142 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:53.409143 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:53.409317 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:24:53.409389 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:53.409391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:53.410077 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:53.410078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:53.410603 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:53.410605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:53.411044 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:53.411045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:53.411120 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:53.411642 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:24:53.411646 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:24:53.413011 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:24:53.414789 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:53.414790 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:53.414791 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:53.414986 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:53.415234 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:53.415235 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:53.415236 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:53.415300 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:53.415605 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:24:53.415606 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:24:53.415658 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 15:24:53.415745 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 15:24:53.415859 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 15:24:53.415968 26666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:53.408914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55548 (local address 127.3.211.253:34867)
0925 15:24:53.408981 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:53.408983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:53.408993 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:24:53.409042 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:53.409043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:53.409057 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:53.409135 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:53.409484 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:53.409989 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:53.410692 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:53.410922 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:53.411031 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:53.413110 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:53.413112 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:53.413113 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:53.413121 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:53.414641 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:53.415080 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:53.415081 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:53.415081 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:53.415150 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:53.415366 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:53.415367 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:53.415367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:53.415494 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:53.415609 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:53.415783 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 15:24:53.415863 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 15:24:53.416445 26587 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.211.252" port: 44569 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:24:53.416870 26589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:53.416764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.252:44569 (local address 127.0.0.1:48690)
0925 15:24:53.416818 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:53.416850 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.252:44569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 15:24:53.417200 26589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:53.417104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.252:44569 (local address 127.0.0.1:48698)
0925 15:24:53.417149 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:53.417173 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.252:44569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 15:24:53.417299 26587 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.211.252:44569: Network error: Client connection negotiation failed: client connection to 127.3.211.252:44569: connect: Connection refused (error 111)
I20260925 15:24:53.426728 26662 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.211.252" port: 44569 } attempt: 1
I20260925 15:24:53.427250 26664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:53.427117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.252:44569 (local address 127.0.0.1:48704)
0925 15:24:53.427182 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:53.427223 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.252:44569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 15:24:53.427577 26664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:53.427485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.252:44569 (local address 127.0.0.1:48718)
0925 15:24:53.427530 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:53.427554 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.252:44569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 15:24:53.427663 26662 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.211.252:44569: Network error: Client connection negotiation failed: client connection to 127.3.211.252:44569: connect: Connection refused (error 111)
I20260925 15:24:53.448555 26587 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.211.252" port: 44569 } attempt: 1
I20260925 15:24:53.449074 26589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:53.448934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.252:44569 (local address 127.0.0.1:48728)
0925 15:24:53.449001 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:53.449044 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.252:44569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 15:24:53.449411 26589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:53.449315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.252:44569 (local address 127.0.0.1:48736)
0925 15:24:53.449361 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:53.449389 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.252:44569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 15:24:53.449503 26587 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.211.252:44569: Network error: Client connection negotiation failed: client connection to 127.3.211.252:44569: connect: Connection refused (error 111)
W20260925 15:24:53.476555 26669 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 15:24:53.476836 26669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:24:53.476867 26669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:24:53.476917 26669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:24:53.478245 26669 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:24:53.478328 26669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:24:53.478353 26669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:24:53.478375 26669 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:24:53.478415 26669 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:24:53.480141 26669 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.211.212:38969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.211.254:37851,127.3.211.253:34867,127.3.211.252:44569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/krb5kdc/kudu_127.3.211.252.keytab
--principal=kudu/127.3.211.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.211.252:44569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.211.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 12:59:25 UTC on ccycloud.quasar-byrsoj.root.comops.site
build id 4671938
I20260925 15:24:53.480527 26669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:24:53.480791 26669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:24:53.485776 26674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:53.485771 26677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:24:53.485786 26675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:24:53.485977 26669 server_base.cc:1034] running on GCE node
I20260925 15:24:53.486333 26669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:24:53.486660 26669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:24:53.487802 26669 hybrid_clock.cc:648] HybridClock initialized: now 1790349893487779 us; error 37 us; skew 500 ppm
Sep 25 15:24:53 dist-test-slave-4bk3 krb5kdc[26506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349893, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:24:53.490396 26669 init.cc:377] Logged in from keytab as kudu/127.3.211.252@KRBTEST.COM (short username kudu)
I20260925 15:24:53.490820 26669 webserver.cc:460] Webserver started at http://127.3.211.252:44089/ using document root <none> and password file <none>
I20260925 15:24:53.491019 26669 fs_manager.cc:359] Metadata directory not provided
I20260925 15:24:53.491060 26669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:24:53.491153 26669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:24:53.492214 26669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/minicluster-data/master-2/data/instance:
uuid: "31a717699d6d487d88ac97cb4806fbb8"
format_stamp: "Formatted at 2026-09-25 15:24:53 on dist-test-slave-4bk3"
server_key: "68b2a2352a84196ccbdd823b61712210"
server_key_iv: "3d0f5e9930ba3096ac9809319cfa7661"
server_key_version: "encryptionkey@0"
I20260925 15:24:53.492596 26669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/minicluster-data/master-2/wal/instance:
uuid: "31a717699d6d487d88ac97cb4806fbb8"
format_stamp: "Formatted at 2026-09-25 15:24:53 on dist-test-slave-4bk3"
server_key: "68b2a2352a84196ccbdd823b61712210"
server_key_iv: "3d0f5e9930ba3096ac9809319cfa7661"
server_key_version: "encryptionkey@0"
I20260925 15:24:53.493911 26669 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:24:53.494737 26684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:24:53.495132 26669 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:24:53.495219 26669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/minicluster-data/master-2/data,/tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/minicluster-data/master-2/wal
uuid: "31a717699d6d487d88ac97cb4806fbb8"
format_stamp: "Formatted at 2026-09-25 15:24:53 on dist-test-slave-4bk3"
server_key: "68b2a2352a84196ccbdd823b61712210"
server_key_iv: "3d0f5e9930ba3096ac9809319cfa7661"
server_key_version: "encryptionkey@0"
I20260925 15:24:53.495280 26669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:24:53.495808 26662 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.211.252" port: 44569 } attempt: 2
I20260925 15:24:53.496294 26664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:53.496148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.252:44569 (local address 127.0.0.1:48752)
0925 15:24:53.496227 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:53.496267 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.252:44569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 15:24:53.496860 26664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:53.496727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.252:44569 (local address 127.0.0.1:48768)
0925 15:24:53.496806 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:53.496838 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.211.252:44569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260925 15:24:53.496999 26662 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.211.252:44569: Network error: Client connection negotiation failed: client connection to 127.3.211.252:44569: connect: Connection refused (error 111)
I20260925 15:24:53.508618 26669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:24:53.510870 26669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:24:53.511006 26669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:24:53.515021 26669 rpc_server.cc:307] RPC server started. Bound to: 127.3.211.252:44569
I20260925 15:24:53.515076 26736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.211.252:44569 every 8 connection(s)
I20260925 15:24:53.515486 26669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/minicluster-data/master-2/data/info.pb
I20260925 15:24:53.516584 26737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:24:53.518244 26737 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.211.254" port: 37851 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:24:53.524642 26587 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.211.252" port: 44569 } attempt: 2
Sep 25 15:24:53 dist-test-slave-4bk3 krb5kdc[26506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349893, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.252@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:24:53.525195 3919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskAzNr2f/build/release/bin/kudu as pid 26669
I20260925 15:24:53.525295 3919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskAzNr2f/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790349610595785-3919-0/minicluster-data/master-2/wal/instance
I20260925 15:24:53.525534 3919 external_mini_cluster.cc:1428] Setting key 4298881f00ae3346e1f7a8114b5b083a
I20260925 15:24:53.531380 26590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:53.519405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55550 (local address 127.3.211.254:37851)
0925 15:24:53.519495 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:53.519497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:53.519735 (+ 238us) server_negotiation.cc:407] Connection header received
0925 15:24:53.520234 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:53.520235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:53.520246 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:53.520318 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:53.521065 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:53.521539 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:53.522540 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:53.522743 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:53.522812 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:53.525703 (+ 2891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:53.525704 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:53.525705 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:53.525713 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:53.527566 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:53.528498 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:53.528499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:53.528500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:53.528616 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:53.529069 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:53.529070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:53.529070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:53.529274 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:53.530251 (+ 977us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:53.530434 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 15:24:53.530651 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 15:24:53.531451 26739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:53.519401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.254:37851 (local address 127.0.0.1:55550)
0925 15:24:53.519594 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:53.519640 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:53.520137 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:53.520469 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:53.520472 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:53.520739 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:24:53.520952 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:53.520957 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:53.521641 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:53.521641 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:53.522452 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:53.522453 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:53.522873 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:53.522875 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:53.523000 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:53.523587 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:24:53.523596 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:24:53.525622 (+ 2026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:24:53.527683 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:53.527692 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:53.527695 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:53.528380 (+ 685us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:53.528712 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:53.528712 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:53.528713 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:53.528961 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:53.529401 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:24:53.529409 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:24:53.529770 (+ 361us) client_negotiation.cc:769] Sending connection context
0925 15:24:53.529921 (+ 151us) client_negotiation.cc:240] Negotiation successful
0925 15:24:53.530356 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":107,"threads_started":1}
I20260925 15:24:53.532546 26737 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.211.253" port: 34867 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:24:53 dist-test-slave-4bk3 krb5kdc[26506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.211.252@KRBTEST.COM
Sep 25 15:24:53 dist-test-slave-4bk3 krb5kdc[26506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349893, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.254@KRBTEST.COM for kudu/127.3.211.252@KRBTEST.COM
I20260925 15:24:53.538393 26589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:53.524964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.252:44569 (local address 127.0.0.1:48772)
0925 15:24:53.525039 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:53.525050 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:53.525091 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:53.525419 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:53.525420 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:53.525596 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:24:53.525646 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:53.525648 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:53.528677 (+ 3029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:53.528679 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:53.530036 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:53.530041 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:53.530655 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:53.530656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:53.530746 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:53.532405 (+ 1659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:24:53.532412 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:24:53.534095 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:24:53.537107 (+ 3012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:53.537108 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:53.537109 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:53.537312 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:53.537575 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:53.537576 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:53.537576 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:53.537643 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:53.538058 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:24:53.538059 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:24:53.538119 (+ 60us) client_negotiation.cc:769] Sending connection context
0925 15:24:53.538171 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 15:24:53.538284 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
Sep 25 15:24:53 dist-test-slave-4bk3 krb5kdc[26506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349893, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.252@KRBTEST.COM for kudu/127.3.211.253@KRBTEST.COM
I20260925 15:24:53.538810 26740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:53.525029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48772 (local address 127.3.211.252:44569)
0925 15:24:53.525235 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:53.525240 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:53.525261 (+ 21us) server_negotiation.cc:407] Connection header received
0925 15:24:53.525286 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:53.525288 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:53.525309 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:53.525409 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:53.525979 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:53.528562 (+ 2583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:53.530181 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:53.530539 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:53.530916 (+ 377us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:53.534202 (+ 3286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:53.534203 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:53.534204 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:53.534210 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:53.537033 (+ 2823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:53.537416 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:53.537421 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:53.537421 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:53.537488 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:53.537722 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:53.537724 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:53.537724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:53.537961 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:53.538560 (+ 599us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:53.538614 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 15:24:53.538695 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":121,"threads_started":1}
I20260925 15:24:53.541658 26587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa0a608c86e445c2bf2f0874c8159805: Bootstrap starting.
I20260925 15:24:53.542059 26666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:53.533664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55556 (local address 127.3.211.253:34867)
0925 15:24:53.533772 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:53.533774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:53.533783 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:24:53.533813 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:53.533814 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:53.533825 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:53.533885 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:53.534370 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:53.535169 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:53.536171 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:53.536502 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:53.536573 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:53.539245 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:53.539246 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:53.539247 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:53.539256 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:53.540868 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:53.541258 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:53.541259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:53.541259 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:53.541321 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:53.541529 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:53.541530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:53.541530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:53.541653 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:53.541727 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:53.541881 (+ 154us) server_negotiation.cc:299] Negotiation successful
0925 15:24:53.541966 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 15:24:53.542061 26739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:53.533290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.253:34867 (local address 127.0.0.1:55556)
0925 15:24:53.533476 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:53.533488 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:53.533538 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:53.533934 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:53.533935 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:53.534170 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:24:53.534252 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:53.534255 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:53.535267 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:53.535269 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:53.536065 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:53.536069 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:53.536638 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:53.536639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:53.536725 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:53.537475 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:24:53.537480 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:24:53.539126 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:24:53.540988 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:53.540989 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:53.540989 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:53.541172 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:53.541389 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:53.541390 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:53.541390 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:53.541453 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:53.541718 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:24:53.541719 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:24:53.541767 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 15:24:53.541855 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 15:24:53.541964 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162}
I20260925 15:24:53.542511 26587 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa0a608c86e445c2bf2f0874c8159805: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:53.542585 26737 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.211.252" port: 44569 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:24:53.542788 26587 log.cc:824] T 00000000000000000000000000000000 P aa0a608c86e445c2bf2f0874c8159805: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:53.543704 26587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa0a608c86e445c2bf2f0874c8159805: No bootstrap required, opened a new log
I20260925 15:24:53.544394 26742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:53.526159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48776 (local address 127.3.211.252:44569)
0925 15:24:53.526319 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:53.526320 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:53.526499 (+ 179us) server_negotiation.cc:407] Connection header received
0925 15:24:53.526641 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:53.526642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:53.526653 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:53.526713 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:53.527879 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:53.528661 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:53.529768 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:53.530243 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:53.530294 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:53.533942 (+ 3648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:53.533953 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:53.533954 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:53.533963 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:53.538022 (+ 4059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:53.539201 (+ 1179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:53.539203 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:53.539203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:53.539272 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:53.543396 (+ 4124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:53.543397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:53.543398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:53.543558 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:53.543738 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:53.544106 (+ 368us) server_negotiation.cc:299] Negotiation successful
0925 15:24:53.544198 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":89,"threads_started":1}
I20260925 15:24:53.546075 26587 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa0a608c86e445c2bf2f0874c8159805 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa0a608c86e445c2bf2f0874c8159805" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 37851 } } peers { permanent_uuid: "d77bf57eb9f74582b5e760b46babade3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 34867 } } peers { permanent_uuid: "31a717699d6d487d88ac97cb4806fbb8" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 44569 } }
I20260925 15:24:53.546264 26587 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa0a608c86e445c2bf2f0874c8159805 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa0a608c86e445c2bf2f0874c8159805, State: Initialized, Role: FOLLOWER
I20260925 15:24:53.546368 26587 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa0a608c86e445c2bf2f0874c8159805 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa0a608c86e445c2bf2f0874c8159805" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 37851 } } peers { permanent_uuid: "d77bf57eb9f74582b5e760b46babade3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 34867 } } peers { permanent_uuid: "31a717699d6d487d88ac97cb4806fbb8" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 44569 } }
I20260925 15:24:53.546823 26587 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa0a608c86e445c2bf2f0874c8159805 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:24:53.547001 26744 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa0a608c86e445c2bf2f0874c8159805 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa0a608c86e445c2bf2f0874c8159805" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 37851 } } peers { permanent_uuid: "d77bf57eb9f74582b5e760b46babade3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 34867 } } peers { permanent_uuid: "31a717699d6d487d88ac97cb4806fbb8" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 44569 } } }
I20260925 15:24:53.547196 26744 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa0a608c86e445c2bf2f0874c8159805 [sys.catalog]: This master's current role is: FOLLOWER
Sep 25 15:24:53 dist-test-slave-4bk3 krb5kdc[26506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349893, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.252@KRBTEST.COM for kudu/127.3.211.252@KRBTEST.COM
W20260925 15:24:53.548877 26755 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aa0a608c86e445c2bf2f0874c8159805: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:24:53.548998 26755 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:24:53.551357 26739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:53.542858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.252:44569 (local address 127.0.0.1:48790)
0925 15:24:53.542890 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:53.542900 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:53.542940 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:53.543430 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:53.543431 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:53.543599 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:24:53.543645 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:53.543647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:53.544593 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:53.544594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:53.545323 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:53.545326 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:53.545844 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:53.545845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:53.545887 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:53.546387 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:24:53.546393 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:24:53.547997 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:24:53.550269 (+ 2272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:53.550270 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:53.550271 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:53.550512 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:53.550691 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:53.550691 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:53.550692 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:53.550743 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:53.551052 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:24:53.551053 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:24:53.551088 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 15:24:53.551145 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 15:24:53.551246 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15}
Sep 25 15:24:53 dist-test-slave-4bk3 krb5kdc[26506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.211.254@KRBTEST.COM
I20260925 15:24:53.551645 26740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:53.542904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48790 (local address 127.3.211.252:44569)
0925 15:24:53.542979 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:53.542981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:53.542991 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:24:53.543014 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:53.543015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:53.543026 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:53.543096 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:53.543736 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:53.544500 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:53.545432 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:53.545732 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:53.545795 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:53.548102 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:53.548103 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:53.548104 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:53.548110 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:53.550165 (+ 2055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:53.550587 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:53.550587 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:53.550588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:53.550632 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:53.550805 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:53.550806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:53.550806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:53.550962 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:53.551030 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:53.551490 (+ 460us) server_negotiation.cc:299] Negotiation successful
0925 15:24:53.551570 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 15:24:53.549985 26519 master_runner.cc:186] Error getting master registration for 127.3.211.253:34867: OK, instance_id { permanent_uuid: "d77bf57eb9f74582b5e760b46babade3" instance_seqno: 1790349893348616 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 15:24:53.553720 26519 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 15:24:53.554222 26737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31a717699d6d487d88ac97cb4806fbb8: Bootstrap starting.
I20260925 15:24:53.555080 26737 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31a717699d6d487d88ac97cb4806fbb8: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:53.555415 26737 log.cc:824] T 00000000000000000000000000000000 P 31a717699d6d487d88ac97cb4806fbb8: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:53.555701 26590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:53.545594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55554 (local address 127.3.211.254:37851)
0925 15:24:53.545634 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:53.545636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:53.545645 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:24:53.545690 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:53.545691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:53.545701 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:53.545752 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:53.546798 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:53.547524 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:53.548553 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:53.548827 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:53.548887 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:53.552121 (+ 3234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:53.552123 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:53.552124 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:53.552133 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:53.554068 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:53.554538 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:53.554540 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:53.554540 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:53.554614 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:53.554862 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:53.554863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:53.554863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:53.555019 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:53.555167 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:53.555509 (+ 342us) server_negotiation.cc:299] Negotiation successful
0925 15:24:53.555591 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 15:24:53.556550 26737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31a717699d6d487d88ac97cb4806fbb8: No bootstrap required, opened a new log
I20260925 15:24:53.558452 26737 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31a717699d6d487d88ac97cb4806fbb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa0a608c86e445c2bf2f0874c8159805" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 37851 } } peers { permanent_uuid: "d77bf57eb9f74582b5e760b46babade3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 34867 } } peers { permanent_uuid: "31a717699d6d487d88ac97cb4806fbb8" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 44569 } }
I20260925 15:24:53.558621 26737 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31a717699d6d487d88ac97cb4806fbb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a717699d6d487d88ac97cb4806fbb8, State: Initialized, Role: FOLLOWER
I20260925 15:24:53.558717 26737 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31a717699d6d487d88ac97cb4806fbb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa0a608c86e445c2bf2f0874c8159805" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 37851 } } peers { permanent_uuid: "d77bf57eb9f74582b5e760b46babade3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 34867 } } peers { permanent_uuid: "31a717699d6d487d88ac97cb4806fbb8" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 44569 } }
I20260925 15:24:53.559055 26758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31a717699d6d487d88ac97cb4806fbb8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa0a608c86e445c2bf2f0874c8159805" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 37851 } } peers { permanent_uuid: "d77bf57eb9f74582b5e760b46babade3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 34867 } } peers { permanent_uuid: "31a717699d6d487d88ac97cb4806fbb8" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 44569 } } }
I20260925 15:24:53.559121 26737 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31a717699d6d487d88ac97cb4806fbb8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:24:53.559202 26758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31a717699d6d487d88ac97cb4806fbb8 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 15:24:53.561350 26769 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 31a717699d6d487d88ac97cb4806fbb8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:24:53.561481 26769 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 25 15:24:53 dist-test-slave-4bk3 krb5kdc[26506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.211.253@KRBTEST.COM
I20260925 15:24:53.562493 26669 master_runner.cc:186] Error getting master registration for 127.3.211.253:34867: OK, instance_id { permanent_uuid: "d77bf57eb9f74582b5e760b46babade3" instance_seqno: 1790349893348616 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 15:24:53.565071 26669 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 15:24:53.565984 26666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:53.556391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55572 (local address 127.3.211.253:34867)
0925 15:24:53.556434 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:53.556435 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:53.556486 (+ 51us) server_negotiation.cc:407] Connection header received
0925 15:24:53.556570 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:53.556570 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:53.556582 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:53.556636 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:53.557270 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:53.557992 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:53.559018 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:53.559322 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:53.559366 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:53.562658 (+ 3292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:53.562660 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:53.562660 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:53.562668 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:53.564586 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:53.565141 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:53.565142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:53.565143 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:53.565213 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:53.565442 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:53.565443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:53.565444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:53.565573 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:53.565755 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:53.565792 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 15:24:53.565869 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 15:24:53.567242 3919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:24:53.568065 26561 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "aa0a608c86e445c2bf2f0874c8159805"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55554
I20260925 15:24:53.568229 26561 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa0a608c86e445c2bf2f0874c8159805 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 15:24:53.568282 26561 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa0a608c86e445c2bf2f0874c8159805 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:53.569093 26561 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa0a608c86e445c2bf2f0874c8159805 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa0a608c86e445c2bf2f0874c8159805" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 37851 } } peers { permanent_uuid: "d77bf57eb9f74582b5e760b46babade3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 34867 } } peers { permanent_uuid: "31a717699d6d487d88ac97cb4806fbb8" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 44569 } }
I20260925 15:24:53.569348 26561 leader_election.cc:288] T 00000000000000000000000000000000 P aa0a608c86e445c2bf2f0874c8159805 [CANDIDATE]: Term 1 election: Requested vote from peers d77bf57eb9f74582b5e760b46babade3 (127.3.211.253:34867), 31a717699d6d487d88ac97cb4806fbb8 (127.3.211.252:44569)
I20260925 15:24:53.569649 26711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aa0a608c86e445c2bf2f0874c8159805" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "31a717699d6d487d88ac97cb4806fbb8"
I20260925 15:24:53.569785 26711 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31a717699d6d487d88ac97cb4806fbb8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:53.570655 26711 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 31a717699d6d487d88ac97cb4806fbb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa0a608c86e445c2bf2f0874c8159805 in term 1.
I20260925 15:24:53.570861 26538 leader_election.cc:302] T 00000000000000000000000000000000 P aa0a608c86e445c2bf2f0874c8159805 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a717699d6d487d88ac97cb4806fbb8, aa0a608c86e445c2bf2f0874c8159805; no voters:
I20260925 15:24:53.571022 26744 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa0a608c86e445c2bf2f0874c8159805 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:24:53.571202 26744 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa0a608c86e445c2bf2f0874c8159805 [term 1 LEADER]: Becoming Leader. State: Replica: aa0a608c86e445c2bf2f0874c8159805, State: Running, Role: LEADER
I20260925 15:24:53.571298 26744 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa0a608c86e445c2bf2f0874c8159805 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa0a608c86e445c2bf2f0874c8159805" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 37851 } } peers { permanent_uuid: "d77bf57eb9f74582b5e760b46babade3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 34867 } } peers { permanent_uuid: "31a717699d6d487d88ac97cb4806fbb8" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 44569 } }
I20260925 15:24:53.571746 26744 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa0a608c86e445c2bf2f0874c8159805 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa0a608c86e445c2bf2f0874c8159805. Latest consensus state: current_term: 1 leader_uuid: "aa0a608c86e445c2bf2f0874c8159805" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa0a608c86e445c2bf2f0874c8159805" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 37851 } } peers { permanent_uuid: "d77bf57eb9f74582b5e760b46babade3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 34867 } } peers { permanent_uuid: "31a717699d6d487d88ac97cb4806fbb8" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 44569 } } }
I20260925 15:24:53.571836 26744 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa0a608c86e445c2bf2f0874c8159805 [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:53.573719 26777 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:24:53.573915 26777 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:24:53.586674 26666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:53.572664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55582 (local address 127.3.211.253:34867)
0925 15:24:53.572724 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:53.572727 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:53.576526 (+ 3799us) server_negotiation.cc:407] Connection header received
0925 15:24:53.576640 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:53.576641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:53.576654 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:53.576746 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:53.577751 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:53.578789 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:53.580101 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:53.580507 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:53.580587 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:53.582484 (+ 1897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:53.582486 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:53.582486 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:53.582494 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:53.584855 (+ 2361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:53.585437 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:53.585439 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:53.585439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:53.585509 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:53.585734 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:53.585735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:53.585736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:53.585892 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:53.585984 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:53.586490 (+ 506us) server_negotiation.cc:299] Negotiation successful
0925 15:24:53.586576 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 15:24:53.588150 26662 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.211.252" port: 44569 } attempt: 3
Sep 25 15:24:53 dist-test-slave-4bk3 krb5kdc[26506](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349893, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.211.253@KRBTEST.COM for kudu/127.3.211.252@KRBTEST.COM
I20260925 15:24:53.596969 26711 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 31a717699d6d487d88ac97cb4806fbb8 [term 1 FOLLOWER]: Refusing update from remote peer aa0a608c86e445c2bf2f0874c8159805: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:24:53.597389 26771 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aa0a608c86e445c2bf2f0874c8159805 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a717699d6d487d88ac97cb4806fbb8" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 44569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:24:53.597708 26664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:53.588486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.211.252:44569 (local address 127.0.0.1:48814)
0925 15:24:53.588548 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0925 15:24:53.588558 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:24:53.588595 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:24:53.588867 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:24:53.588868 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:24:53.589471 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:24:53.589525 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:53.589528 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:53.590423 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:53.590424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:53.591254 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:24:53.591256 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:24:53.591811 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:24:53.591812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:24:53.591866 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:53.592428 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:24:53.592435 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:24:53.594300 (+ 1865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:24:53.596453 (+ 2153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:53.596454 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:53.596455 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:53.596711 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:53.596939 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:24:53.596940 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:24:53.596940 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:24:53.597002 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:24:53.597363 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:24:53.597364 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:24:53.597417 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 15:24:53.597488 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 15:24:53.597602 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 15:24:53.598080 26742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:53.588572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48814 (local address 127.3.211.252:44569)
0925 15:24:53.588619 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:53.588642 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:53.588651 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:24:53.588678 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:53.588679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:53.588690 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:53.588814 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:53.589610 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:53.590335 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:53.591366 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:53.591699 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:53.591753 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:53.594427 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:53.594428 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:53.594429 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:53.594437 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:53.596247 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:53.596797 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:53.596799 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:53.596799 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:53.596866 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:53.597125 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:53.597126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:53.597126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:53.597296 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:53.597378 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:53.597902 (+ 524us) server_negotiation.cc:299] Negotiation successful
0925 15:24:53.597991 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 15:24:53.599417 26758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31a717699d6d487d88ac97cb4806fbb8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa0a608c86e445c2bf2f0874c8159805. Latest consensus state: current_term: 1 leader_uuid: "aa0a608c86e445c2bf2f0874c8159805" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa0a608c86e445c2bf2f0874c8159805" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 37851 } } peers { permanent_uuid: "d77bf57eb9f74582b5e760b46babade3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 34867 } } peers { permanent_uuid: "31a717699d6d487d88ac97cb4806fbb8" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 44569 } } }
I20260925 15:24:53.599547 26758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31a717699d6d487d88ac97cb4806fbb8 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:24:53.601199 26662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d77bf57eb9f74582b5e760b46babade3: Bootstrap starting.
I20260925 15:24:53.601969 26662 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d77bf57eb9f74582b5e760b46babade3: Neither blocks nor log segments found. Creating new log.
I20260925 15:24:53.602313 26662 log.cc:824] T 00000000000000000000000000000000 P d77bf57eb9f74582b5e760b46babade3: Log is configured to *not* fsync() on all Append() calls
I20260925 15:24:53.603605 26662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d77bf57eb9f74582b5e760b46babade3: No bootstrap required, opened a new log
I20260925 15:24:53.606585 26590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:53.572941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55566 (local address 127.3.211.254:37851)
0925 15:24:53.572988 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:53.572990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:53.576725 (+ 3735us) server_negotiation.cc:407] Connection header received
0925 15:24:53.576752 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:53.576752 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:53.576763 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:53.576817 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:53.585145 (+ 8328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:53.590205 (+ 5060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:53.591373 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:53.591683 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:53.591749 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:53.594517 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:53.594519 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:53.594519 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:53.594527 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:53.602746 (+ 8219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:53.604426 (+ 1680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:53.604428 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:53.604429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:53.604516 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:53.605572 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:53.605574 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:53.605574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:53.605758 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:53.606334 (+ 576us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:53.606380 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 15:24:53.606468 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 15:24:53.609120 26662 raft_consensus.cc:348] T 00000000000000000000000000000000 P d77bf57eb9f74582b5e760b46babade3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa0a608c86e445c2bf2f0874c8159805" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 37851 } } peers { permanent_uuid: "d77bf57eb9f74582b5e760b46babade3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 34867 } } peers { permanent_uuid: "31a717699d6d487d88ac97cb4806fbb8" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 44569 } }
I20260925 15:24:53.609314 26662 raft_consensus.cc:729] T 00000000000000000000000000000000 P d77bf57eb9f74582b5e760b46babade3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d77bf57eb9f74582b5e760b46babade3, State: Initialized, Role: FOLLOWER
I20260925 15:24:53.609412 26662 consensus_queue.cc:260] T 00000000000000000000000000000000 P d77bf57eb9f74582b5e760b46babade3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa0a608c86e445c2bf2f0874c8159805" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 37851 } } peers { permanent_uuid: "d77bf57eb9f74582b5e760b46babade3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 34867 } } peers { permanent_uuid: "31a717699d6d487d88ac97cb4806fbb8" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 44569 } }
I20260925 15:24:53.609824 26636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "aa0a608c86e445c2bf2f0874c8159805" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d77bf57eb9f74582b5e760b46babade3"
I20260925 15:24:53.609889 26635 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d77bf57eb9f74582b5e760b46babade3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:24:53.611027 26635 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d77bf57eb9f74582b5e760b46babade3 [term 1 FOLLOWER]: Refusing update from remote peer aa0a608c86e445c2bf2f0874c8159805: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:24:53.611277 26740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:53.573136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48804 (local address 127.3.211.252:44569)
0925 15:24:53.573195 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:53.573197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:53.590292 (+ 17095us) server_negotiation.cc:407] Connection header received
0925 15:24:53.590314 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:53.590315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:53.590328 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:53.590381 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:53.592597 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:53.593251 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:53.594783 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:53.595349 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:53.595426 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:53.605559 (+ 10133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:24:53.605562 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:24:53.605562 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:24:53.605571 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:24:53.608625 (+ 3054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:53.609698 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:53.609700 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:53.609700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:53.610004 (+ 304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:24:53.610271 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:24:53.610272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:24:53.610272 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:24:53.610408 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:24:53.610524 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:53.611025 (+ 501us) server_negotiation.cc:299] Negotiation successful
0925 15:24:53.611166 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 15:24:53.611382 26771 consensus_queue.cc:1035] T 00000000000000000000000000000000 P aa0a608c86e445c2bf2f0874c8159805 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d77bf57eb9f74582b5e760b46babade3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 34867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:24:53.612629 26636 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d77bf57eb9f74582b5e760b46babade3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa0a608c86e445c2bf2f0874c8159805 in term 1.
I20260925 15:24:53.609810 26662 sys_catalog.cc:564] T 00000000000000000000000000000000 P d77bf57eb9f74582b5e760b46babade3 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 15:24:53.615909 26799 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d77bf57eb9f74582b5e760b46babade3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260925 15:24:53.616396 26781 mvcc.cc:205] Tried to move back new op lower bound from 7333273164126457856 to 7333273164068847616. Current Snapshot: MvccSnapshot[applied={T|T < 7333273164126457856}]
I20260925 15:24:53.617197 26758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31a717699d6d487d88ac97cb4806fbb8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aa0a608c86e445c2bf2f0874c8159805" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa0a608c86e445c2bf2f0874c8159805" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 37851 } } peers { permanent_uuid: "d77bf57eb9f74582b5e760b46babade3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 34867 } } peers { permanent_uuid: "31a717699d6d487d88ac97cb4806fbb8" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 44569 } } }
I20260925 15:24:53.617316 26758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31a717699d6d487d88ac97cb4806fbb8 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:24:53.618268 26771 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa0a608c86e445c2bf2f0874c8159805 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aa0a608c86e445c2bf2f0874c8159805" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa0a608c86e445c2bf2f0874c8159805" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 37851 } } peers { permanent_uuid: "d77bf57eb9f74582b5e760b46babade3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 34867 } } peers { permanent_uuid: "31a717699d6d487d88ac97cb4806fbb8" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 44569 } } }
I20260925 15:24:53.618384 26771 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa0a608c86e445c2bf2f0874c8159805 [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:53.618672 26782 mvcc.cc:205] Tried to move back new op lower bound from 7333273164126457856 to 7333273164068847616. Current Snapshot: MvccSnapshot[applied={T|T < 7333273164126457856}]
W20260925 15:24:53.616034 26799 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:24:53.619449 26785 sys_catalog.cc:455] T 00000000000000000000000000000000 P d77bf57eb9f74582b5e760b46babade3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa0a608c86e445c2bf2f0874c8159805" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa0a608c86e445c2bf2f0874c8159805" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 37851 } } peers { permanent_uuid: "d77bf57eb9f74582b5e760b46babade3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 34867 } } peers { permanent_uuid: "31a717699d6d487d88ac97cb4806fbb8" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 44569 } } }
I20260925 15:24:53.619590 26785 sys_catalog.cc:458] T 00000000000000000000000000000000 P d77bf57eb9f74582b5e760b46babade3 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:24:53.619642 26785 sys_catalog.cc:455] T 00000000000000000000000000000000 P d77bf57eb9f74582b5e760b46babade3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa0a608c86e445c2bf2f0874c8159805. Latest consensus state: current_term: 1 leader_uuid: "aa0a608c86e445c2bf2f0874c8159805" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa0a608c86e445c2bf2f0874c8159805" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 37851 } } peers { permanent_uuid: "d77bf57eb9f74582b5e760b46babade3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 34867 } } peers { permanent_uuid: "31a717699d6d487d88ac97cb4806fbb8" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 44569 } } }
I20260925 15:24:53.619688 26785 sys_catalog.cc:458] T 00000000000000000000000000000000 P d77bf57eb9f74582b5e760b46babade3 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:24:53.620532 26777 catalog_manager.cc:1347] Generated new cluster ID: 49efc11b8a804c1ba4859c60d2c033f4
I20260925 15:24:53.620586 26777 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:24:53.623881 26744 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa0a608c86e445c2bf2f0874c8159805 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "aa0a608c86e445c2bf2f0874c8159805" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa0a608c86e445c2bf2f0874c8159805" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 37851 } } peers { permanent_uuid: "d77bf57eb9f74582b5e760b46babade3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 34867 } } peers { permanent_uuid: "31a717699d6d487d88ac97cb4806fbb8" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 44569 } } }
I20260925 15:24:53.623991 26744 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa0a608c86e445c2bf2f0874c8159805 [sys.catalog]: This master's current role is: LEADER
I20260925 15:24:53.624411 26787 mvcc.cc:205] Tried to move back new op lower bound from 7333273164126457856 to 7333273164068847616. Current Snapshot: MvccSnapshot[applied={T|T < 7333273164126457856}]
I20260925 15:24:53.624621 26786 sys_catalog.cc:455] T 00000000000000000000000000000000 P d77bf57eb9f74582b5e760b46babade3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "aa0a608c86e445c2bf2f0874c8159805" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa0a608c86e445c2bf2f0874c8159805" member_type: VOTER last_known_addr { host: "127.3.211.254" port: 37851 } } peers { permanent_uuid: "d77bf57eb9f74582b5e760b46babade3" member_type: VOTER last_known_addr { host: "127.3.211.253" port: 34867 } } peers { permanent_uuid: "31a717699d6d487d88ac97cb4806fbb8" member_type: VOTER last_known_addr { host: "127.3.211.252" port: 44569 } } }
I20260925 15:24:53.624708 26786 sys_catalog.cc:458] T 00000000000000000000000000000000 P d77bf57eb9f74582b5e760b46babade3 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:24:53.642169 26777 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:24:53.642980 26777 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:24:53.662346 26777 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa0a608c86e445c2bf2f0874c8159805: Generated new TSK 0
I20260925 15:24:53.662885 26777 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 15:24:53.694885 26666 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:24:53.692740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55594 (local address 127.3.211.253:34867)
0925 15:24:53.692792 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:53.692794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:53.693857 (+ 1063us) server_negotiation.cc:407] Connection header received
0925 15:24:53.694113 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:53.694114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:53.694126 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:53.694157 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:53.694810 (+ 653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55594: BlockingRecv error: recv got EOF from 127.0.0.1:55594 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260925 15:24:53.695434 26590 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:24:53.692997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55580 (local address 127.3.211.254:37851)
0925 15:24:53.693037 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:53.693039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:53.693761 (+ 722us) server_negotiation.cc:407] Connection header received
0925 15:24:53.694229 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:53.694230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:53.694242 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:53.694271 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:53.695356 (+ 1085us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55580: BlockingRecv error: recv got EOF from 127.0.0.1:55580 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 15:24:53.696100 26740 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:24:53.693201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48818 (local address 127.3.211.252:44569)
0925 15:24:53.693240 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:53.693242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:53.694056 (+ 814us) server_negotiation.cc:407] Connection header received
0925 15:24:53.695661 (+ 1605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:53.695663 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:53.695675 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:53.695706 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:53.696038 (+ 332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48818: BlockingRecv error: recv got EOF from 127.0.0.1:48818 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 15:24:53.705271 26666 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:24:53.700651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55596 (local address 127.3.211.253:34867)
0925 15:24:53.700697 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:53.700699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:53.704583 (+ 3884us) server_negotiation.cc:407] Connection header received
0925 15:24:53.704723 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:53.704725 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:53.704750 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:53.704783 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:53.705207 (+ 424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55596: BlockingRecv error: recv got EOF from 127.0.0.1:55596 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 15:24:53.709208 26666 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:24:53.708053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55598 (local address 127.3.211.253:34867)
0925 15:24:53.708096 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:53.708300 (+ 204us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:53.708624 (+ 324us) server_negotiation.cc:407] Connection header received
0925 15:24:53.708731 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:53.708732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:53.708744 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:53.708775 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:53.709149 (+ 374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55598: BlockingRecv error: recv got EOF from 127.0.0.1:55598 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 15:24:53.713421 26666 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:24:53.712564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55608 (local address 127.3.211.253:34867)
0925 15:24:53.712604 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:53.712606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:53.712835 (+ 229us) server_negotiation.cc:407] Connection header received
0925 15:24:53.712937 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:53.712938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:53.712949 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:53.712978 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:53.713347 (+ 369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55608: BlockingRecv error: recv got EOF from 127.0.0.1:55608 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 15:24:53.719656 26666 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:24:53.718781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55610 (local address 127.3.211.253:34867)
0925 15:24:53.718825 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:53.718827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:53.719073 (+ 246us) server_negotiation.cc:407] Connection header received
0925 15:24:53.719170 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:53.719171 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:53.719183 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:53.719213 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:53.719597 (+ 384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55610: BlockingRecv error: recv got EOF from 127.0.0.1:55610 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 15:24:53.730273 26666 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:24:53.729302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55618 (local address 127.3.211.253:34867)
0925 15:24:53.729358 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:53.729360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:53.729641 (+ 281us) server_negotiation.cc:407] Connection header received
0925 15:24:53.729756 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:53.729757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:53.729769 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:53.729800 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:53.730206 (+ 406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55618: BlockingRecv error: recv got EOF from 127.0.0.1:55618 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260925 15:24:53.748908 26666 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:24:53.747844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55626 (local address 127.3.211.253:34867)
0925 15:24:53.747904 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:53.747906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:53.748189 (+ 283us) server_negotiation.cc:407] Connection header received
0925 15:24:53.748323 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:53.748360 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:53.748374 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:53.748407 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:53.748843 (+ 436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55626: BlockingRecv error: recv got EOF from 127.0.0.1:55626 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260925 15:24:53.783196 26666 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:24:53.782377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55628 (local address 127.3.211.253:34867)
0925 15:24:53.782424 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:53.782427 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:53.782570 (+ 143us) server_negotiation.cc:407] Connection header received
0925 15:24:53.782630 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:53.782631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:53.782642 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:53.782724 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:53.783134 (+ 410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55628: BlockingRecv error: recv got EOF from 127.0.0.1:55628 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260925 15:24:53.849514 26666 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:24:53.848615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55636 (local address 127.3.211.253:34867)
0925 15:24:53.848695 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:53.848698 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:53.848864 (+ 166us) server_negotiation.cc:407] Connection header received
0925 15:24:53.848895 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:53.848896 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:53.848908 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:53.848959 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:53.849454 (+ 495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55636: BlockingRecv error: recv got EOF from 127.0.0.1:55636 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260925 15:24:53.979964 26666 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:24:53.979180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55642 (local address 127.3.211.253:34867)
0925 15:24:53.979255 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:53.979257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:53.979366 (+ 109us) server_negotiation.cc:407] Connection header received
0925 15:24:53.979398 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:53.979399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:53.979410 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:53.979469 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:53.979882 (+ 413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55642: BlockingRecv error: recv got EOF from 127.0.0.1:55642 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260925 15:24:54.237932 26666 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:24:54.237287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55654 (local address 127.3.211.253:34867)
0925 15:24:54.237339 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:54.237362 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:54.237389 (+ 27us) server_negotiation.cc:407] Connection header received
0925 15:24:54.237416 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:54.237417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:54.237436 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:54.237500 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:24:54.237838 (+ 338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55654: BlockingRecv error: recv got EOF from 127.0.0.1:55654 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 15:24:54.562003 26769 catalog_manager.cc:1259] Loaded cluster ID: 49efc11b8a804c1ba4859c60d2c033f4
I20260925 15:24:54.562093 26769 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 31a717699d6d487d88ac97cb4806fbb8: loading cluster ID for follower catalog manager: success
I20260925 15:24:54.562970 26769 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 31a717699d6d487d88ac97cb4806fbb8: acquiring CA information for follower catalog manager: success
I20260925 15:24:54.563329 26769 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 31a717699d6d487d88ac97cb4806fbb8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 15:24:54.619242 26799 catalog_manager.cc:1259] Loaded cluster ID: 49efc11b8a804c1ba4859c60d2c033f4
I20260925 15:24:54.619333 26799 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d77bf57eb9f74582b5e760b46babade3: loading cluster ID for follower catalog manager: success
I20260925 15:24:54.620318 26799 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d77bf57eb9f74582b5e760b46babade3: acquiring CA information for follower catalog manager: success
I20260925 15:24:54.620785 26799 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d77bf57eb9f74582b5e760b46babade3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 15:24:54.753934 26869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:54.751230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55660 (local address 127.3.211.253:34867)
0925 15:24:54.751377 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:54.751379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:54.751389 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:24:54.751446 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:54.751446 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:54.751496 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:54.751571 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:24:54.751788 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:54.752299 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:54.753118 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:54.753329 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:54.753401 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:54.753552 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:24:54.753662 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:24:54.753724 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:54.753779 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 15:24:54.753846 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":75,"threads_started":1}
I20260925 15:24:54.758445 26875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:54.755914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48820 (local address 127.3.211.252:44569)
0925 15:24:54.756034 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:54.756036 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:54.756044 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:24:54.756080 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:54.756080 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:54.756115 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:54.756185 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:24:54.756254 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:54.756792 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:54.757613 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:54.757825 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:54.757896 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:54.758019 (+ 123us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:24:54.758127 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:24:54.758193 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:54.758291 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 15:24:54.758314 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":57,"threads_started":1}
I20260925 15:24:54.763638 26875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:54.760514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48828 (local address 127.3.211.252:44569)
0925 15:24:54.760559 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:54.760561 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:54.760909 (+ 348us) server_negotiation.cc:407] Connection header received
0925 15:24:54.760937 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:54.760938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:54.760948 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:54.761013 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:24:54.761163 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:54.761634 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:54.762735 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:54.763044 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:54.763106 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:54.763291 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:24:54.763321 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:24:54.763363 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:54.763523 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 15:24:54.763550 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 15:24:54.763705 26869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:54.760612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55672 (local address 127.3.211.253:34867)
0925 15:24:54.760654 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:54.760657 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:54.760718 (+ 61us) server_negotiation.cc:407] Connection header received
0925 15:24:54.760791 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:54.760791 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:54.760801 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:54.760875 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:24:54.761010 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:54.761819 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:54.762821 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:54.763044 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:54.763123 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:54.763268 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:24:54.763305 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:24:54.763352 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:54.763416 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 15:24:54.763580 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 15:24:54.763825 26882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:24:54.760475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55596 (local address 127.3.211.254:37851)
0925 15:24:54.760830 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0925 15:24:54.760832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:24:54.760840 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:24:54.760899 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:24:54.760899 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:24:54.760937 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:24:54.761041 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:24:54.761139 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:54.761870 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:54.762735 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:24:54.763045 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:24:54.763106 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:24:54.763330 (+ 224us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:24:54.763463 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:24:54.763518 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 15:24:54.763690 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 15:24:54.763714 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":47,"threads_started":1}
I20260925 15:24:54.766083 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 26519
I20260925 15:24:54.771327 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 26594
I20260925 15:24:54.776306 3919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskAzNr2f/build/release/bin/kudu with pid 26669
2026-09-25T15:24:54Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2912 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5861 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (284186 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST